{"id":4352,"date":"2024-03-14T01:22:02","date_gmt":"2024-03-14T01:22:02","guid":{"rendered":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/"},"modified":"2025-07-31T06:35:19","modified_gmt":"2025-07-31T06:35:19","slug":"how-to-control-the-time-properties-of-data-in-apache-beam","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/","title":{"rendered":"Apache Beam Timestamps &#038; Watermarks Guide"},"content":{"rendered":"<p>In Apache Beam, you can use Timestamps and Watermarks provided by the Apache Beam SDK to control the temporal properties of data. Timestamps are used to specify the timestamp of data elements, while Watermarks are used to control the progress of the data stream.<\/p>\n<p>To control the temporal properties of data, you can use the ParDo function in the data processing pipeline to specify the timestamp of data elements. For example, you can use the WithTimestamps function to set the timestamp for data elements.<\/p>\n<pre class=\"post-pre\"><code>PCollection&lt;MyData&gt; myData = ... <span class=\"hljs-comment\">\/\/ \u83b7\u53d6\u6570\u636e\u96c6<\/span>\r\n\r\nPCollection&lt;MyData&gt; timestampedData = myData.apply(ParDo.of(<span class=\"hljs-keyword\">new<\/span> <span class=\"hljs-title class_\">DoFn<\/span>&lt;MyData, MyData&gt;() {\r\n    <span class=\"hljs-meta\">@ProcessElement<\/span>\r\n    <span class=\"hljs-keyword\">public<\/span> <span class=\"hljs-keyword\">void<\/span> <span class=\"hljs-title function_\">processElement<\/span><span class=\"hljs-params\">(ProcessContext c)<\/span> {\r\n        <span class=\"hljs-type\">MyData<\/span> <span class=\"hljs-variable\">data<\/span> <span class=\"hljs-operator\">=<\/span> c.element();\r\n        <span class=\"hljs-type\">Instant<\/span> <span class=\"hljs-variable\">timestamp<\/span> <span class=\"hljs-operator\">=<\/span> ... <span class=\"hljs-comment\">\/\/ \u6307\u5b9a\u65f6\u95f4\u6233<\/span>\r\n        c.outputWithTimestamp(data, timestamp);\r\n    }\r\n}));\r\n<\/code><\/pre>\n<p>After specifying the timestamp of the data elements, you can also use Window operators to allocate data into windows in order to control the time attributes of the data stream. For example, you can use the FixedWindows function to allocate data elements into fixed size time windows.<\/p>\n<pre class=\"post-pre\"><code>PCollection&lt;MyData&gt; windowedData = timestampedData.apply(Window.into(FixedWindows.of(Duration.standardMinutes(<span class=\"hljs-number\">1<\/span>))));\r\n<\/code><\/pre>\n<p>Finally, Watermarks can be used to control the progress of data streams. They indicate the current progress of the data stream, and Apache Beam will use Watermarks to control the processing and triggering of data. The generation logic of Watermarks can be specified by setting the WatermarkEvaluator function.<\/p>\n<pre class=\"post-pre\"><code>PCollection&lt;MyData&gt; input = ... <span class=\"hljs-comment\">\/\/ \u8f93\u5165\u6570\u636e\u96c6<\/span>\r\n\r\nPCollection&lt;MyData&gt; output = input.apply(WithTimestamps.of(<span class=\"hljs-keyword\">new<\/span> <span class=\"hljs-title class_\">MyTimestampFunction<\/span>()))\r\n                                    .apply(Window.into(FixedWindows.of(Duration.standardMinutes(<span class=\"hljs-number\">1<\/span>))));\r\n\r\nPTransform&lt;PCollection&lt;MyData&gt;, PCollection&lt;MyResult&gt;&gt; transform = ... <span class=\"hljs-comment\">\/\/ \u5b9a\u4e49\u6570\u636e\u5904\u7406\u8f6c\u6362<\/span>\r\n\r\nPCollection&lt;MyResult&gt; finalOutput = output.apply(transform);\r\n\r\npipeline.run();\r\n<\/code><\/pre>\n<p>By using the above methods, it is possible to flexibly control the time attributes of data in Apache Beam, enabling more precise data processing and windowing operations.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In Apache Beam, you can use Timestamps and Watermarks provided by the Apache Beam SDK to control the temporal properties of data. Timestamps are used to specify the timestamp of data elements, while Watermarks are used to control the progress of the data stream. To control the temporal properties of data, you can use the [&hellip;]<\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[1],"tags":[907,342,3824,1319,1320],"class_list":["post-4352","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-apache-beam","tag-data-processing","tag-time-properties","tag-timestamps","tag-watermarks"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Apache Beam Timestamps &amp; Watermarks Guide - Blog - Silicon Cloud<\/title>\n<meta name=\"description\" content=\"Learn how to control time properties in Apache Beam using Timestamps and Watermarks. Master temporal data processing with our comprehensive guide.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Beam Timestamps &amp; Watermarks Guide\" \/>\n<meta property=\"og:description\" content=\"Learn how to control time properties in Apache Beam using Timestamps and Watermarks. Master temporal data processing with our comprehensive guide.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-14T01:22:02+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-07-31T06:35:19+00:00\" \/>\n<meta name=\"author\" content=\"Sophia Anderson\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:site\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sophia Anderson\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/\"},\"author\":{\"name\":\"Sophia Anderson\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/19a24313de9c988db3d69226b4a40a30\"},\"headline\":\"Apache Beam Timestamps &#038; Watermarks Guide\",\"datePublished\":\"2024-03-14T01:22:02+00:00\",\"dateModified\":\"2025-07-31T06:35:19+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/\"},\"wordCount\":211,\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"keywords\":[\"Apache Beam\",\"Data Processing\",\"Time Properties\",\"Timestamps\",\"Watermarks\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/\",\"name\":\"Apache Beam Timestamps & Watermarks Guide - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\"},\"datePublished\":\"2024-03-14T01:22:02+00:00\",\"dateModified\":\"2025-07-31T06:35:19+00:00\",\"description\":\"Learn how to control time properties in Apache Beam using Timestamps and Watermarks. Master temporal data processing with our comprehensive guide.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.silicloud.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Apache Beam Timestamps &#038; Watermarks Guide\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"name\":\"Silicon Cloud Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\",\"name\":\"Silicon Cloud Blog\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"contentUrl\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"width\":1024,\"height\":1024,\"caption\":\"Silicon Cloud Blog\"},\"image\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\",\"https:\/\/twitter.com\/SiliCloudGlobal\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/19a24313de9c988db3d69226b4a40a30\",\"name\":\"Sophia Anderson\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c726c09aa40e37115fb5c62d0c3ed62c16ca255d3763e2e3ae83a70ddf8c2175?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c726c09aa40e37115fb5c62d0c3ed62c16ca255d3763e2e3ae83a70ddf8c2175?s=96&d=mm&r=g\",\"caption\":\"Sophia Anderson\"},\"url\":\"https:\/\/www.silicloud.com\/blog\/author\/sophiaanderson\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Apache Beam Timestamps & Watermarks Guide - Blog - Silicon Cloud","description":"Learn how to control time properties in Apache Beam using Timestamps and Watermarks. Master temporal data processing with our comprehensive guide.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/","og_locale":"en_US","og_type":"article","og_title":"Apache Beam Timestamps & Watermarks Guide","og_description":"Learn how to control time properties in Apache Beam using Timestamps and Watermarks. Master temporal data processing with our comprehensive guide.","og_url":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/","og_site_name":"Blog - Silicon Cloud","article_publisher":"https:\/\/www.facebook.com\/SiliCloudGlobal\/","article_published_time":"2024-03-14T01:22:02+00:00","article_modified_time":"2025-07-31T06:35:19+00:00","author":"Sophia Anderson","twitter_card":"summary_large_image","twitter_creator":"@SiliCloudGlobal","twitter_site":"@SiliCloudGlobal","twitter_misc":{"Written by":"Sophia Anderson","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/#article","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/"},"author":{"name":"Sophia Anderson","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/19a24313de9c988db3d69226b4a40a30"},"headline":"Apache Beam Timestamps &#038; Watermarks Guide","datePublished":"2024-03-14T01:22:02+00:00","dateModified":"2025-07-31T06:35:19+00:00","mainEntityOfPage":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/"},"wordCount":211,"publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"keywords":["Apache Beam","Data Processing","Time Properties","Timestamps","Watermarks"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/","url":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/","name":"Apache Beam Timestamps & Watermarks Guide - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/#website"},"datePublished":"2024-03-14T01:22:02+00:00","dateModified":"2025-07-31T06:35:19+00:00","description":"Learn how to control time properties in Apache Beam using Timestamps and Watermarks. Master temporal data processing with our comprehensive guide.","breadcrumb":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/blog\/how-to-control-the-time-properties-of-data-in-apache-beam\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.silicloud.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Apache Beam Timestamps &#038; Watermarks Guide"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/blog\/#website","url":"https:\/\/www.silicloud.com\/blog\/","name":"Silicon Cloud Blog","description":"","publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.silicloud.com\/blog\/#organization","name":"Silicon Cloud Blog","url":"https:\/\/www.silicloud.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","contentUrl":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","width":1024,"height":1024,"caption":"Silicon Cloud Blog"},"image":{"@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/SiliCloudGlobal\/","https:\/\/twitter.com\/SiliCloudGlobal"]},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/19a24313de9c988db3d69226b4a40a30","name":"Sophia Anderson","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c726c09aa40e37115fb5c62d0c3ed62c16ca255d3763e2e3ae83a70ddf8c2175?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c726c09aa40e37115fb5c62d0c3ed62c16ca255d3763e2e3ae83a70ddf8c2175?s=96&d=mm&r=g","caption":"Sophia Anderson"},"url":"https:\/\/www.silicloud.com\/blog\/author\/sophiaanderson\/"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/4352","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/comments?post=4352"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/4352\/revisions"}],"predecessor-version":[{"id":149006,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/4352\/revisions\/149006"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/media?parent=4352"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/categories?post=4352"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/tags?post=4352"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}