{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":477925242,"defaultBranch":"master","name":"spark","ownerLogin":"anishshri-db","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2022-04-05T00:49:09.000Z","ownerAvatar":"https://github.com/avatars/u/100322362?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1727486501.0","currentOid":""},"activityList":{"items":[{"before":null,"after":"7205d315e2bd177ec86c67a05d5a83bf079f8b70","ref":"refs/heads/task/SPARK-49823","pushedAt":"2024-09-28T01:21:41.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"[SPARK-49823] Avoid flush during shutdown in rocksdb close path","shortMessageHtmlLink":"[SPARK-49823] Avoid flush during shutdown in rocksdb close path"}},{"before":null,"after":"21f3e8c3dde2ca2aa2d361c09322148d0953931b","ref":"refs/heads/task/SPARK-49802","pushedAt":"2024-09-27T01:24:37.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"[SPARK-49802] Add support for read change feed for map types used in stateful processors","shortMessageHtmlLink":"[SPARK-49802] Add support for read change feed for map types used in …"}},{"before":"80d6651cf6a1835d0de3e12e08253d2a9816d499","after":"339dd5b93316fecd0455b53b2cedee2b5333a184","ref":"refs/heads/master","pushedAt":"2024-09-26T20:45:56.000Z","pushType":"push","commitsCount":11,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"[SPARK-49791][SQL] Make DelegatingCatalogExtension more extendable\n\n### What changes were proposed in this pull request?\n\nThis PR updates `DelegatingCatalogExtension` so that it's more extendable\n- `initialize` becomes not final, so that sub-classes can overwrite it\n- `delegate` becomes `protected`, so that sub-classes can access it\n\nIn addition, this PR fixes a mistake that `DelegatingCatalogExtension` is just a convenient default implementation, it's actually the `CatalogExtension` interface that indicates this catalog implementation will delegate requests to the Spark session catalog. https://github.com/apache/spark/pull/47724 should use `CatalogExtension` instead.\n\n### Why are the changes needed?\n\nUnblock the Iceberg extension.\n\n### Does this PR introduce _any_ user-facing change?\n\nno\n\n### How was this patch tested?\n\nexisting tests\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nno\n\nCloses #48257 from cloud-fan/catalog.\n\nLead-authored-by: Wenchen Fan <wenchen@databricks.com>\nCo-authored-by: Wenchen Fan <cloud0fan@gmail.com>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","shortMessageHtmlLink":"[SPARK-49791][SQL] Make DelegatingCatalogExtension more extendable"}},{"before":"b24f3ef70bfb12293bea223fa2383a6431677d08","after":"fe12722fe01772cc468a5918f4c15c4d2a3d9945","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-25T21:34:48.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Misc fix","shortMessageHtmlLink":"Misc fix"}},{"before":"66be13988166b1e08d9b7d30996fa040b6d5a56b","after":"b24f3ef70bfb12293bea223fa2383a6431677d08","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-25T20:19:28.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Address Jungtaek's comments","shortMessageHtmlLink":"Address Jungtaek's comments"}},{"before":"08429d00261872f15a211ec3ddfc106f61eff0f5","after":"66be13988166b1e08d9b7d30996fa040b6d5a56b","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-25T19:53:38.000Z","pushType":"push","commitsCount":60,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Merge branch 'master' into task/SPARK-49656","shortMessageHtmlLink":"Merge branch 'master' into task/SPARK-49656"}},{"before":"55d0233d19cc52bee91a9619057d9b6f33165a0a","after":"80d6651cf6a1835d0de3e12e08253d2a9816d499","ref":"refs/heads/master","pushedAt":"2024-09-25T19:25:15.000Z","pushType":"push","commitsCount":20,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"[SPARK-48195][FOLLOWUP] Accumulator reset() no longer needed in CollectMetricsExec.doExecute()\n\n### What changes were proposed in this pull request?\n\nSmall followup to https://github.com/apache/spark/pull/48037.\n`collector.reset()` is no longer needed in `CollectMetricsExec.doExecute()` because it is reset in `resetMetrics()`. This doesn't really matter in practice, but removing to clean up.\n\n### Why are the changes needed?\n\nTiny cleanup.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nThis change doesn't matter in practice. Just cleanup.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nNo.\n\nCloses #48243 from juliuszsompolski/SPARK-48195-followup.\n\nAuthored-by: Julek Sompolski <Juliusz Sompolski>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","shortMessageHtmlLink":"[SPARK-48195][FOLLOWUP] Accumulator reset() no longer needed in Colle…"}},{"before":"a5fe21ac225ab8a7b21a688fa5ebb3884e2c16b1","after":"16f91f875bc68b50ffa06c400576744867996d7b","ref":"refs/heads/task/SPARK-49745","pushedAt":"2024-09-25T02:34:52.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"HeartSaVioR","name":"Jungtaek Lim","path":"/HeartSaVioR","primaryAvatarUrl":"https://github.com/avatars/u/1317309?s=80&v=4"},"commit":{"message":"Update sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceTransformWithStateSuite.scala","shortMessageHtmlLink":"Update sql/core/src/test/scala/org/apache/spark/sql/execution/datasou…"}},{"before":"781600415b6cf937e4b223b06a8fbfd4aaf00647","after":"a5fe21ac225ab8a7b21a688fa5ebb3884e2c16b1","ref":"refs/heads/task/SPARK-49745","pushedAt":"2024-09-24T20:51:49.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Misc fix","shortMessageHtmlLink":"Misc fix"}},{"before":"1316d04988eb3fdead3257faa34f0fddd9d44cdc","after":"781600415b6cf937e4b223b06a8fbfd4aaf00647","ref":"refs/heads/task/SPARK-49745","pushedAt":"2024-09-24T20:11:55.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Misc fix","shortMessageHtmlLink":"Misc fix"}},{"before":"c0b5d78429332e0d8182b7d850d2adadc70866f4","after":"1316d04988eb3fdead3257faa34f0fddd9d44cdc","ref":"refs/heads/task/SPARK-49745","pushedAt":"2024-09-24T15:21:16.000Z","pushType":"push","commitsCount":40,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Merge branch 'master' into task/SPARK-49745","shortMessageHtmlLink":"Merge branch 'master' into task/SPARK-49745"}},{"before":"d4665fa1df716305acb49912d41c396b39343c93","after":"55d0233d19cc52bee91a9619057d9b6f33165a0a","ref":"refs/heads/master","pushedAt":"2024-09-24T15:05:52.000Z","pushType":"push","commitsCount":39,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"[SPARK-49713][PYTHON][FOLLOWUP] Make function `count_min_sketch` accept long seed\n\n### What changes were proposed in this pull request?\nMake function `count_min_sketch` accept long seed\n\n### Why are the changes needed?\nexisting implementation only accepts int seed, which is inconsistent with other `ExpressionWithRandomSeed`:\n\n```py\nIn [3]:     >>> from pyspark.sql import functions as sf\n   ...:     >>> spark.range(100).select(\n   ...:     ...     sf.hex(sf.count_min_sketch(\"id\", sf.lit(1.5), 0.6, 1111111111111111111))\n   ...:     ... ).show(truncate=False)\n\n...\nAnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve \"count_min_sketch(id, 1.5, 0.6, 1111111111111111111)\" due to data type mismatch: The 4th parameter requires the \"INT\" type, however \"1111111111111111111\" has the type \"BIGINT\". SQLSTATE: 42K09;\n'Aggregate [unresolvedalias('hex(count_min_sketch(id#64L, 1.5, 0.6, 1111111111111111111, 0, 0)))]\n+- Range (0, 100, step=1, splits=Some(12))\n...\n\n```\n\n### Does this PR introduce _any_ user-facing change?\nno\n\n### How was this patch tested?\nadded doctest\n\n### Was this patch authored or co-authored using generative AI tooling?\nno\n\nCloses #48223 from zhengruifeng/count_min_sk_long_seed.\n\nAuthored-by: Ruifeng Zheng <ruifengz@apache.org>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","shortMessageHtmlLink":"[SPARK-49713][PYTHON][FOLLOWUP] Make function <code>count_min_sketch</code> acce…"}},{"before":"7e9a821f050ed650b43db9746980ca9f265de6e2","after":"c5a540b8ce20bb72e116ee5f74ff5c2bf351b5a9","ref":"refs/heads/task/SPARK-49630","pushedAt":"2024-09-24T06:49:24.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Test fix","shortMessageHtmlLink":"Test fix"}},{"before":"4e77d0cc732c82d39e65ad947d795f6ffbdf2197","after":"7e9a821f050ed650b43db9746980ca9f265de6e2","ref":"refs/heads/task/SPARK-49630","pushedAt":"2024-09-24T06:39:13.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Misc fix","shortMessageHtmlLink":"Misc fix"}},{"before":"da5abe5f5b93da6c6b80b2e61946a8ff51df663c","after":"4e77d0cc732c82d39e65ad947d795f6ffbdf2197","ref":"refs/heads/task/SPARK-49630","pushedAt":"2024-09-24T01:34:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Address Jungtaek's comments","shortMessageHtmlLink":"Address Jungtaek's comments"}},{"before":"ba63f90a92959923506f35e5cb0ca39f9dfe2ddc","after":"c0b5d78429332e0d8182b7d850d2adadc70866f4","ref":"refs/heads/task/SPARK-49745","pushedAt":"2024-09-24T00:49:47.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Add test","shortMessageHtmlLink":"Add test"}},{"before":"41125d57e9c11281da7bc2cabad2aab61c37578c","after":"ba63f90a92959923506f35e5cb0ca39f9dfe2ddc","ref":"refs/heads/task/SPARK-49745","pushedAt":"2024-09-24T00:39:55.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Address review comments","shortMessageHtmlLink":"Address review comments"}},{"before":"6d28e569ca60643d60ca52a9fcf55dc29bbf4393","after":"41125d57e9c11281da7bc2cabad2aab61c37578c","ref":"refs/heads/task/SPARK-49745","pushedAt":"2024-09-23T05:07:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Misc fix","shortMessageHtmlLink":"Misc fix"}},{"before":null,"after":"6d28e569ca60643d60ca52a9fcf55dc29bbf4393","ref":"refs/heads/task/SPARK-49745","pushedAt":"2024-09-23T04:56:45.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"[SPARK-49745] Add change to read registered timers through state data source reader","shortMessageHtmlLink":"[SPARK-49745] Add change to read registered timers through state data…"}},{"before":"e0d6b849fdf6bb89f4bb17c9b477cd08c234872c","after":"08429d00261872f15a211ec3ddfc106f61eff0f5","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-20T06:15:51.000Z","pushType":"push","commitsCount":15,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Merge branch 'master' into task/SPARK-49656","shortMessageHtmlLink":"Merge branch 'master' into task/SPARK-49656"}},{"before":"f3c8d26eb0c3fd7f77950eb08c70bb2a9ab6493c","after":"d4665fa1df716305acb49912d41c396b39343c93","ref":"refs/heads/master","pushedAt":"2024-09-20T06:11:18.000Z","pushType":"push","commitsCount":14,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"[SPARK-49677][SS] Ensure that changelog files are written on commit and forceSnapshot flag is also reset\n\n### What changes were proposed in this pull request?\nEnsure that changelog files are written on commit and forceSnapshot flag is also reset\n\n### Why are the changes needed?\nWithout these changes, we are not writing the changelog files per batch and we are also trying to upload full snapshot each time since the flag is not being reset correctly\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nAdded unit tests\n\nBefore:\n```\n[info] Run completed in 3 seconds, 438 milliseconds.\n[info] Total number of tests run: 1\n[info] Suites: completed 1, aborted 0\n[info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0\n[info] *** 1 TEST FAILED ***\n```\n\nAfter:\n```\n[info] Run completed in 4 seconds, 155 milliseconds.\n[info] Total number of tests run: 1\n[info] Suites: completed 1, aborted 0\n[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0\n[info] All tests passed.\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\nNo\n\nCloses #48125 from anishshri-db/task/SPARK-49677.\n\nAuthored-by: Anish Shrigondekar <anish.shrigondekar@databricks.com>\nSigned-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>","shortMessageHtmlLink":"[SPARK-49677][SS] Ensure that changelog files are written on commit a…"}},{"before":"18150f0721c6e7b5b18dfbb9821dceedc657a114","after":"e0d6b849fdf6bb89f4bb17c9b477cd08c234872c","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-19T03:47:49.000Z","pushType":"push","commitsCount":29,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Merge branch 'master' into task/SPARK-49656","shortMessageHtmlLink":"Merge branch 'master' into task/SPARK-49656"}},{"before":"8d78f5b02c256423ee125c31bf746a1a15dbcf25","after":"f3c8d26eb0c3fd7f77950eb08c70bb2a9ab6493c","ref":"refs/heads/master","pushedAt":"2024-09-19T03:45:50.000Z","pushType":"push","commitsCount":28,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Revert \"[SPARK-49422][CONNECT][SQL] Add groupByKey to sql/api\"\n\nThis reverts commit af45902d33c4d8e38a6427ac1d0c46fe057bb45a.","shortMessageHtmlLink":"Revert \"[SPARK-49422][CONNECT][SQL] Add groupByKey to sql/api\""}},{"before":"3e67d44dabb6ca73c8782e4b06c1cb5a9889ff07","after":"18150f0721c6e7b5b18dfbb9821dceedc657a114","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-18T23:16:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Fix test","shortMessageHtmlLink":"Fix test"}},{"before":"519de7ebd83f92b388e94b8bd123d833d27cb860","after":"3e67d44dabb6ca73c8782e4b06c1cb5a9889ff07","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-18T20:59:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Add comments","shortMessageHtmlLink":"Add comments"}},{"before":"207d3874771555ca14839e96588ea7b8410432fc","after":"519de7ebd83f92b388e94b8bd123d833d27cb860","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-18T20:50:50.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Misc fix","shortMessageHtmlLink":"Misc fix"}},{"before":"034b90e55975ae820279c887b027c10ab0040bda","after":"207d3874771555ca14839e96588ea7b8410432fc","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-18T20:49:39.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Misc fix","shortMessageHtmlLink":"Misc fix"}},{"before":"31af97d839480ce0972cfb996d3b7374c81d7471","after":"034b90e55975ae820279c887b027c10ab0040bda","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-18T20:36:29.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Add test","shortMessageHtmlLink":"Add test"}},{"before":null,"after":"31af97d839480ce0972cfb996d3b7374c81d7471","ref":"refs/heads/task/SPARK-49656","pushedAt":"2024-09-18T17:56:14.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"[SPARK-49656] Add support for state variables with collection types and read change feed options","shortMessageHtmlLink":"[SPARK-49656] Add support for state variables with collection types a…"}},{"before":"d086f700a2522ad9aa383ce74e5b89b786406da8","after":"da5abe5f5b93da6c6b80b2e61946a8ff51df663c","ref":"refs/heads/task/SPARK-49630","pushedAt":"2024-09-17T22:50:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anishshri-db","name":"Anish","path":"/anishshri-db","primaryAvatarUrl":"https://github.com/avatars/u/100322362?s=80&v=4"},"commit":{"message":"Address Jing's comments","shortMessageHtmlLink":"Address Jing's comments"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0yOFQwMToyMTo0MS4wMDAwMDBazwAAAATDIh1S","endCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0xN1QyMjo1MDoyNS4wMDAwMDBazwAAAAS5DdwT"}},"title":"Activity · anishshri-db/spark"}