Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fail to read string list on certain line #1092

Closed
printfCalvin opened this issue Dec 2, 2022 · 0 comments · Fixed by #1144
Closed

fail to read string list on certain line #1092

printfCalvin opened this issue Dec 2, 2022 · 0 comments · Fixed by #1144
Assignees

Comments

@printfCalvin
Copy link
Contributor

Kuzu failed to read the string list on line 5. TypeUtils::toString(col->readValue(j)) prints [,,,,,,,,,,,,,,,,,,,,,] in the test file. I suspect that this issue is related to copying strings in memory or in page. More specifically, the error could occur in InMemOverflowFile::copyString(const char* rawString, PageByteCursor& overflowCursor) in in_mem_file.pp.

My csv file:

id,feature1,feature2,feature3,feature4,feature5,feature6
0,"[4,54,61,73,1,26,59,62,35,83,20,4,66,62,41,9,31,95,46,5,53,17,77,45,48,53,36,86,33,58,22,87,38,84,46,17]","[76.78378139439906,98.24132490111909,96.93881604049189,61.3326820546709,4.42606328646209,0.40551441584074643,13.397252704913388,94.10022713958341,30.28605620290723,36.614560166042644,89.81962445391883,31.436380495645068,54.89821840124055,43.603095762412224,6.499417612685054,58.45462257019302,84.40678976619022,15.64189183874064,22.42989686860415,41.28702077148407,3.6924869058629017,49.660349861654815,81.79828644296319,65.78909387591091,53.34837356527721,85.51257407736,14.968813497140154,56.72354768631239,37.41748177480889]","[false,false,false,true,false,false,true,true,true,true,false,true,false,false,false,true,true,true,false,false,false,true,true,true,true,false,false,false,true,true,false,false,true,true,true,false,true,true]","[1346-07-13,1754-02-13,1390-01-01,1746-08-05,1692-09-27,1864-04-17,1717-07-22,1090-02-09,1238-09-23,1702-06-29,1967-12-02,1662-05-18,1575-10-21,1194-11-17,1603-07-08,1776-12-31,1839-08-25,1501-04-22,1773-03-25,1556-09-12,1703-03-23,2007-05-03,1237-12-29,1802-01-07,1633-07-13,1141-08-11,1595-12-18,1050-09-12,1004-01-24,1629-07-23,1768-07-26,1093-03-21,1072-08-05,1501-11-21,1133-02-17,1218-03-17,1156-02-12,1873-01-27,1651-10-22,1696-04-09,1199-12-17,1658-05-10,1626-08-04]","[1719-04-29 22:08:36,1492-09-09 17:25:44,2001-10-05 10:59:52,1471-05-15 15:35:19,1751-04-20 3:41:13,1340-09-10 7:45:12,1047-03-24 16:29:18,1922-06-04 21:42:43,1825-11-04 19:54:22,1687-07-02 0:35:51,1167-01-20 23:05:47,1624-12-07 20:25:44,1353-11-24 17:14:16,1127-10-29 1:06:34,1790-05-22 18:24:14,1151-08-04 23:33:05,1409-10-20 3:54:11,1685-03-06 3:23:26,1854-04-14 9:36:57,1275-07-27 23:39:37,1262-01-12 1:54:50,1279-06-02 10:14:30,1137-10-22 16:25:30,1442-11-04 21:18:02,1665-09-14 20:42:58,1997-12-24 19:51:21,1607-11-24 10:44:34,1797-06-03 5:00:14,1857-09-23 8:44:37,1900-05-31 18:47:10,1011-02-14 5:01:48,1875-04-20 9:48:11,1548-02-13 20:54:43,1936-04-17 4:47:37,1607-12-10 19:07:51,1079-04-23 8:21:07,1517-01-31 10:57:53,1277-12-06 15:05:31,1405-06-30 11:12:01,1422-01-10 12:09:22,1728-07-14 21:47:34,1580-09-22 10:35:58,1490-03-30 8:22:31,1397-09-15 11:34:56,1641-04-23 20:28:43,1842-11-18 11:31:04,1989-12-11 12:57:19,1526-11-03 19:13:34,1581-09-23 21:51:39]","[AUKeMUtbLuPueVzNxsMlpktgJYdoKVeAmKKCZLb,mvKmHZKWXudQSLdcKxzuMOkXkh,YwkbUIGZwfzCKwdjCTENCUCMlwBvJGDymgBPjcl,KBYxJydMdktod,lgOKxltczMWBewWmFkkzZWDXvcllLTAjGPSzgz,pgBUOtUs,PgwHNEMpIsaQkRySogGOWEgKzLnswemNJHb,zZcbP,vlUdwuOEYkbnismppyNILMAJWbia,bdRXXlhS]"
1,"[8,1,84,55,94,84,23,79,53,6,70,21,6,89,48,74,58,58,42,23,4,51,80,58]","[95.26044630210201,71.29702595217846,66.59641718160826,13.904308725382053,25.318753105457237,9.114152536348785,1.330347473625182,27.802919654977522,90.96668401725518,30.84739755152728,76.69035874644443,53.70731213717516,28.776380200233266,61.83131173748924,33.46381620299881,0.3507171390896957,25.96631767988734,93.31388902193956,51.74727099213956,87.55755909514366,14.59356525022204,5.934299255822717,5.660976923121785,6.0332310544770955,21.095833581370115,1.4630572601545988,14.663104940801352,86.03957123406173,84.19317187287572,15.091008587500564,2.6921573088481643]","[false,true,true,true,true,false,false,false,true,false,true,false,true,true,false,false,true,true,false,true,true,false,true,true,true,false,true,true,false,false,true,false,true]","[1749-03-01,1069-06-30,1545-08-27,1875-07-06,1116-09-15,2001-10-08,1051-07-26,1183-03-03,1047-09-05,1014-01-31,1525-07-05,1624-10-21,1317-11-22,1946-05-23,1685-07-04,1812-09-02,1950-03-21,1130-01-17,1147-09-29,1097-05-13,1525-10-12,1654-07-23,1614-01-05,1446-11-25,1254-12-30,1181-10-16,1904-08-29,1573-09-29,1779-12-24,2002-01-01,1775-09-17,1276-05-13,1977-04-21]","[1346-08-19 17:15:54,1548-07-25 5:14:11,1740-02-21 14:21:27,1984-11-16 19:54:46,1952-05-02 9:15:42,1273-10-03 23:41:44,1768-03-19 17:30:27,1222-05-16 20:29:25,1849-04-09 12:54:54,1102-07-22 15:45:47,1654-02-20 2:07:46,1300-05-18 22:30:27,1651-12-05 4:17:31,1245-03-07 8:48:26,1907-01-07 4:59:13,1479-05-22 7:10:41,1371-11-14 3:43:07,1695-08-07 10:00:04,1490-07-01 0:07:04,1967-10-11 7:02:32,2002-11-17 21:02:40,1068-12-15 9:45:23,1407-05-21 11:01:22,1165-10-22 21:23:13,1600-01-28 10:18:37,1133-01-24 2:53:46,1231-12-06 21:03:03,1105-06-01 19:40:29,1762-08-23 1:55:21,1010-02-18 12:29:08,1790-07-25 6:14:15,1693-05-09 21:34:58,1121-12-29 10:06:57,1166-05-19 18:04:27,1684-05-12 19:33:54,1565-04-14 21:31:06,1637-02-15 23:56:26,1908-07-04 13:38:07,1707-10-30 16:41:43,1428-11-04 13:29:07,1222-10-15 10:49:05,1325-07-16 11:20:09,1821-09-29 11:13:33,1433-05-30 23:20:56,1791-03-03 21:48:48,1858-06-23 18:13:59,1718-11-09 13:04:00]","[tlpoxMkHXbIKZcdkAWtkXmbkDucGpc,jR,DXwbAORwqbjqcjPRQphGTEXXswuJBXSv,sBdgEQUTxFUBEwekLmJWusmaULBUXcWUkkYExzh,pYmKeFdJqFfXVQNSfpPlkCzTJ,RBSoBAmoUSxET,WZmZKQVdU,SYlmfUzORIaMGoYSTKuuFLOlZZEpyzDGgA,HUfpdgUerprUSvWsIxiwXKOYiSeaWXCPhzNNbkM,ApEDekfvP,XjOdCSVHcQdcCHAJIt,KRxsLiDUvweqTsPwSJgqOyuDBzkrhjJNkaCWZWx,rkumROQiddWLkKIJjdKswDWxtqem,ePeVKpCYIRpKNrJRsdGBQUv,MSdxvEFsIFaxqLLPElYoyBuRrgvhfg,BplIRWlXaGJsOkvKm,UShkuavEmCdMXEKibJsGlHLgjFOZVtKtdykjyP,E,AMKOObbLkSXrzmLGMTEypvIoxKLc,aZjainrpVeFAUdCxSaOnlNKnSWwbsjHYMFyfi,etQlFXTEsWdEscxefwHBchRJPZkicYo,detYcAxvGMNNsyqyaJIaDoJeyIS,rQIcduLQDtukkwFgVxoudOt,G,hVVTxdiKbrowIbMtIRy,OzNCuUaQismwwOGsASgCjoyIvRRtpLRBbZe,yXSnkdfxxgRlVQM,PebA,SeJQBFqhbIgZmKYN,WWyPq,uMhtT,pCiEQYuWsmBdBiB,jWxjKUXZZsQKOOGJz,FzTgqdkzIRgjHU,rohdZYdSAzPGScX,iBtdoBu,CNIBbMnkQUnecNjMrWu,HMwPdRWIiWKnzqFBK,RCwAuJouIzcbWSFDkcCWpcELke,SPTPgIVPXkInMxNX]"
2,"[91,100,6,66,59,30,18,90,85,59]","[7.186582513873718,78.06106332475412,24.41047317939593,10.045618606958516,17.286599166310314,77.56671881324458,17.125525425620005,61.047661479969975,7.744874405572522,3.9929755904786535,80.38398697543006,9.407164956334524,81.66812486934865,76.64702078136473,84.74443335883585,10.091133027595255,83.45642569752606,7.740183902462661,59.47423127380612,59.900203036967625,21.009668521489388,76.4245255791442,77.26681538445688,20.056324534904725,95.15534612565524,57.62348987690932,87.22113274784105,65.68412848779157,37.30378982781798,53.026083316581065,31.66325180905245,50.1041245011997,20.881020990344503,43.569565713565396,53.3486265286923,82.03678314928736,58.26824050129326,74.13878090653859,65.7491173196038,76.67915821116414,34.44202617027915]","[true,false,true,true,true,true,false,true,false,true,false,false,false,true,true,false]","[1063-05-09,1491-11-17,1364-12-08,1620-11-22,1571-10-23,1745-01-09,1092-02-19,2015-03-07,1377-03-08,1974-06-21,1491-03-14,2020-01-13,1570-10-08,1357-05-08,1930-04-11]","[]","[otpbnlmahqqEhwZfLQJ,RugzABWALYTXdUwvaGMgBIKGdM,hFKkyZeqhiSTQrwtMIYpQgBBvWny,NBhhD,uvYjKcbHSEOqBVqjiztYdKxKDITxEaKs,GZQrlMnGYN,emfEvuWCv,xTkTgRIc,QUKqfxMMUtKxiXDfPYsUqDZBYefGwFCwzP,xNoScnyYcKg,TnLv,ATaDAaeWrybVWoSBBlHLVcSGmYvAUzPI,lHefgcoULaWdWAuKUrRSDNQGvKXeWIk,EPCCByNCpZDRrJjiibUSw,cajYYNfOzgClMcMV,cgdIYQkpRyveVWIAhnWSEyMMGwDGBLLY,FNvvJwIOenisietueBiXDy,HJeAFpWkIdOkpARSzVzZqjCtCAXLdzQcy,ZQTfb,BJflzbRyGkYCjSuCMhEbPRzxEH,DVcpHfCXiNmEClttFrOFtkrdAuetAh,hZWSpwoIeJJDgRumGPsiJZrWemTVw,HAntcBKYHPZKnzUPbNZa,WvrGQcfGaAzZ,cd,gvIrEI,IerAzhYMIIliZSLkOkKXiMyUgFutfKMyOoxuV,YNephKMHGxZGSFaPVFSpQMpBMwoyMGmTHyDdP,EKNSDLQqrgYHsVyBXEECWSBbprXlRpBhAMH,vRwCdthzfsNoEmwkkarMamYDuvPry,aurDSsu,AczwDdNNQktqzPaWst,SJ,CHfyvWMyrxThcySCIFmkkUVjEjLuCoYwhfJcWQC,rOKzEflOkZACq,lKxWWtgKjJBstVLfwPmlEd,GLUcGHjTQTwsYbzElOzrxisdfWZcO,vkUOrXvcfASxYKAXKJSTHRqlUIiVxuoMJrVSEgWW,lwMUSKsFBrbysZRQQzYKVvYLQVeLyvkTdmRNnuVK,zxCXMheYhWLxPmPJXMGGI,ll,vBLUsAqgDkxgwSxzquXTInaBPSLSlpguLC,bFrSBIwHNRJHvJSYqnAXJdeQYnci,DJhfLsDVXD,JciaopbLOSNz,xTGaSeYqMphxkQpTDXAbOgdPtWuHSRIdeG,zDosDHXDfiqccbSdYQxCJF]"
3,"[24,72,9,51,18,93,19,39,52,75,97,8,62,13,83,89,59,50,3,21,96,64,68,54,93,23,63,29,84,77,32,59,97,35,25,33,46,69,93]","[86.22453463621208,53.43940785777151,68.8576294166873,13.001476395072197,49.4475190505269,2.529301803001671,21.565892432304157,80.84112970208548,41.77938003022713,16.820986638720548,42.090635170382726,40.1697539384928,82.82336536722057,19.75706471733649]","[false,false,true,false,false,false,true,false,false,true,false,false,true,true,false,false,false,false,false,false,true,false,true,true,true,true,false,true,true]","[1531-02-13,1067-12-14,1005-01-04,1867-04-27,1323-06-27,1171-12-19,1058-01-02,1801-07-17,1893-12-14,1323-05-06,1545-02-11,1702-02-22,1418-11-15,1742-12-22,1606-02-17,1025-12-24,1169-08-17,1449-03-12,1335-11-13,1629-05-17,1114-02-08,1297-03-10,1087-07-05,2008-09-23,1386-12-19,1013-09-16,1083-11-24,1633-03-24,1653-07-13,1272-02-09,1289-11-24,1289-01-14,1014-07-25,1092-09-11,1266-05-30,1700-08-02,1725-03-17,1776-09-24,1435-09-04,1394-03-03,1095-12-08,1338-12-07,1082-07-16]","[1012-04-25 2:37:25,1543-10-18 11:57:24,1544-07-03 7:47:14,1822-05-12 23:51:04,1095-10-27 1:58:06,1161-07-05 9:23:56,1968-07-16 12:30:33,1508-02-01 13:43:15]","[SyBCmYinMTiPNZgZCvHMxeaAEOARqISVnJ,HatK,KDhBZydMaXxzkaiKntkBgT,eIFuxNmgDDCMlzrMIzSqykP,uPghpGQnlpkkfCuATtF,ECMiUxiKbCGFPQqVBsMVNjrVGIVcSMakrQVcz,egtYmmJvp,zMQnWWCPvpdVBDiUwvrlRlNCwKMZRgxjrti,NGBJFrFQ,xHoWyeSXbpVqUerMqMgmgmlPU,eulTYuXLJopvzGvpXHOkrIarfxa,YCXgSeIXwAptqJkcvWBZSqDeLlEzExQLU,FGiUNyajkoqPSSK,bSTIXLuBcaeovBkOm,diWb,wXauHDvxhiPsWzyUMFSkVVlhrHE,iWAzpbgHltfpSHP,EIdvEQcFAma,mrPGEwFoAofmMlLLKE,zcaeRzOrfqj,IYSvZZmkGlIKIzWFEQpyvPaiqpxrUjkfHg,zLwbAReTVKDaxAzKWkzHctpcoA]"
4,"[77,2,71,14,60,26,64,26,42,73,46,41,28,41,65,11,99,80,87,14,56,7,34,33,100,46,70,76,27,55,93,90]","[6.063713937384685,14.961578807311371,37.211026652615416,97.11400954233798,50.088141729467516,75.90481814201519,14.618305422523736,91.09746083916495,5.657845733624622,21.59883923957634,99.73205155010076,6.100214916500879,12.256408099081028,11.407332452812092,3.823204401375102,69.2344631768466,41.57842901613827,17.159511253368407,4.229991290839397,97.34307074429007,16.739476286088596]","[false,true,false,true,true,true,true,false]","[1668-06-29,1477-03-13,1908-10-22,1984-12-07,1153-09-06,1616-10-19,1922-12-07,1786-08-10,1380-06-22,1327-01-03,1611-06-01,1887-03-15,1998-05-20,1827-08-22,1764-02-08,1804-10-15,1346-03-05,1910-12-07,1249-01-25,1497-09-24,1444-07-07,1183-07-07,1613-10-18,1845-07-24,1192-03-07,1181-05-18,1604-05-27,1204-09-12,1106-03-09,1037-07-06,1058-01-07,1939-10-22,1665-02-04,1846-01-10,1691-02-07,1712-02-16,1520-11-12,1521-08-20,1464-04-04,2007-12-03,1450-10-09,1489-01-31,1031-01-06,1366-02-18,1465-01-16]","[1604-12-03 20:08:04,1699-09-10 22:34:26,1858-09-05 15:50:38,1052-02-22 4:31:35,1413-02-02 18:38:44,1653-05-02 2:33:30,1798-02-25 15:23:02,1463-11-04 13:35:19,1485-07-03 22:03:41,1924-06-11 9:08:23,1193-05-05 15:18:40,1045-09-20 22:33:27,1618-08-26 3:22:56,1140-03-18 8:50:30,1959-05-21 4:16:32,1276-07-04 23:41:28,1172-09-13 5:20:07,1116-10-20 20:06:49,1058-05-03 11:25:33,1045-10-27 14:38:54,1766-07-05 11:21:16,1714-01-10 11:01:05,1692-04-25 6:48:44,1063-07-10 18:12:17,1764-11-10 23:48:09,1773-06-13 2:01:25,1809-03-26 16:14:45]","[aZ,gMoQTTQJSbWngFUjcCPttHFrtAZ,olfXNAqKtfgCJbUboR,BdnuQcWJIVhRcrKkIOQwnWNtrjGmUkiuxjG,cqCWFlSyIgiGdeNtDQnrBgBffOrtqtWNWJEHWA,rAlEOiRAToMyFOtAHRnBxGDMVRFbFzJo,omBUPqsZSMtkSFnBwuMhgn,OjBbFQpnHSwXkkmZibPkKDIhxSJnsEDI,OGGqsMtQgpqCydZLPNHeAjVzwbPvlgvudjJAq,siMoMuOXffnMFc,vsEmDyCAMWQKSBmMTUmpjoAmzA,SnENlvQOfWxvfaiQKFecXgNRdJkXYfDvYxzydD,OpbmchJHZSKnvAAkEyaghswjHizSJWSOWGp,tWdytuqnvfiDuAQsAKPYsVGxRyzOdtV,zPXRmhUjfXWPORiNZfNMVHDfakIErr,JZjWxLfyCjVN,yTeBW,fhWAbfKuBiQIprkiccBWXATdIcaolHxnfGO]"

schema:

create node table arrow_list (id INT64, feature1 INT64[], feature2 DOUBLE[], feature3 BOOLEAN[], feature4 DATE[], feature5 TIMESTAMP[], feature6 STRING[], PRIMARY KEY (id));

copy_csv.cypher:

COPY arrow_list FROM "dataset/copy-csv-arrow-node-test/list.csv" (HEADER=true);

test code:

#include "test/test_utility/include/test_helper.h"
#include <iostream>
#include <fstream>

using namespace std;
using namespace kuzu::common;
using namespace kuzu::storage;
using namespace kuzu::testing;

class CopyCSVArrowNodeTest : public InMemoryDBTest {
    string getInputCSVDir() override { return "dataset/copy-csv-arrow-node-test/"; }

public:
    static bool CheckEquals(const vector<string>& expected, const Literal& listVal) {
        if (listVal.dataType.typeID != LIST) {
            return false;
        }
        if (expected.size() != listVal.listVal.size()) {
            return false;
        }
        for (auto i = 0u; i < expected.size(); i++) {
            if (expected[i] != TypeUtils::toString(listVal.listVal[i])) {
                cout << expected[i] << " vs " << TypeUtils::toString(listVal.listVal[i]) << endl;
                return false;
            }
        }
        return true;
    }

    void checkElement(DataType type, string& element, Literal cell) {
        switch (type.typeID) {
            case INT64: {
                EXPECT_EQ(stoll(element), cell.val.int64Val);
            }
                break;
            case DOUBLE: {
                EXPECT_EQ(stod(element), cell.val.doubleVal);
            }
                break;
            case BOOL: {
                element[0] = toupper(element[0]);
                EXPECT_EQ(element, TypeUtils::toString(cell));
            }
                break;
            case STRING: {
                EXPECT_EQ(element, cell.strVal);
            }
                break;
            case DATE: {
                vector<string> splitDate = CopyCSVArrowNodeTest::splitStr(element, "-");
                auto d = Date::FromDate(stoi(splitDate[0]), stoi(splitDate[1]), stoi(splitDate[2]));
                EXPECT_EQ(d, cell.val.dateVal);
            }
                break;
            case TIMESTAMP: {
                int pos = element.find(' ');
                string  date = element.substr(0, pos);
                string time = element.substr(pos + 1);
                vector<string> splitDate = CopyCSVArrowNodeTest::splitStr(date, "-");
                auto d = Date::FromDate(stoi(splitDate[0]), stoi(splitDate[1]), stoi(splitDate[2]));

                vector<string> splitTime = CopyCSVArrowNodeTest::splitStr(time, ":");
                auto t = Time::FromTime( stoi(splitTime[0]), stoi(splitTime[1]), stoi(splitTime[2]));
                EXPECT_EQ(Timestamp::FromDatetime(d, t), cell.val.timestampVal);
            }
                break;
            case INTERVAL: {
                EXPECT_EQ(element, Interval::toString(cell.val.intervalVal));
            }
                break;
            case LIST: {
                int64_t from = 1, to = element.length() - 2;
                const char delimiter = ',';
                vector<pair<int64_t, int64_t>> splitElement;
                int bracket = 0;
                int64_t  last = from;
                for (int64_t i = from; i <= to; i++) {
                    if (element[i] == '[') {
                        bracket += 1;
                    } else if (element[i] == ']') {
                        bracket -= 1;
                    } else if (bracket == 0 && element[i] == delimiter) {
                        if (i - last == 0) {
                            last = i + 1;
                            continue;
                        }
                        splitElement.emplace_back(make_pair(last, i - last));
                        last = i + 1;
                    }
                }
                if (to - last + 1 != 0) {
                    splitElement.emplace_back(make_pair(last, to - last + 1));
                }

                    cout << "size: " << splitElement.size() << " " << "elements: " ;
                    for (auto& p : splitElement) {
                        string x = element.substr(p.first, p.second);
                        cout << x << ",";
                    }
                    cout << endl;
                    cout << TypeUtils::toString(cell) << endl << endl;

                EXPECT_EQ(splitElement.size(), cell.listVal.size());
                for (int i = 0; i < splitElement.size(); ++i) {
                    auto list_e = cell.listVal.at(i);
                    string temp = element.substr(splitElement[i].first, splitElement[i].second);

                    checkElement(list_e.dataType, temp, list_e);
                }
            }
                break;
            default:
                break;
        }
    }


    static vector<string> splitStr(string& s, string delimiter) {
        size_t pos = 0;
        std::string token;
        vector<string> split;
        while ((pos = s.find(delimiter)) != std::string::npos) {
            token = s.substr(0, pos);
            split.emplace_back(token);
            s.erase(0, pos + delimiter.length());
        }
        split.emplace_back(s);

        return split;
    }

    static vector<string> splitStrList(string& l, const char delimiter) {
        vector<string> split;
        int bracket = 0, last = 0;
        for (int i = 0; i < l.length(); i++) {
            if (l[i] == '"') {
                bracket += 1;
            } else if (bracket % 2 == 0 && l[i] == delimiter) {
                split.emplace_back(l.substr(last, i - last));
                last = i + 1;
            }
        }
        split.emplace_back(l.substr(last, l.length() - last));

//        cout << "size: " << split.size() << " " << "elements: " ;
//        for (auto& p : split) {
//            cout << p << ", ";
//        }
//        cout << endl;

        for (int i = 1; i < split.size(); ++i) {
            split[i] = split[i].substr(1, split[i].length() - 2);
        }

        return split;
    }
};

TEST_F(CopyCSVArrowNodeTest, ListTest) {
    auto storageManager = getStorageManager(*database);
    auto catalog = getCatalog(*database);
    auto tableID = catalog->getReadOnlyVersion()->getNodeTableIDFromName("arrow_list");

    ifstream MyReadFile("dataset/copy-csv-arrow-node-test/list.csv");
    string line;
    vector<string> lines;
    bool header = true;

    if (header) {
        getline (MyReadFile, line);
    }
    while (getline(MyReadFile, line)) {
        lines.emplace_back(line);
    }

    for (int j = 0; j < lines.size(); j++) {
        vector<string> split = CopyCSVArrowNodeTest::splitStrList(lines[j], ',');

        for (int i = 0; i < split.size(); i++) {
            string name;
            if (i == 0) {
                name = "id";
            } else {
                name = "feature" + to_string(i);
            }
            auto propertyIdx = catalog->getReadOnlyVersion()->getNodeProperty(
                    tableID, name);
            auto col =
                    storageManager->getNodesStore().getNodePropertyColumn(tableID, propertyIdx.propertyID);

            auto type = col->readValue(j).dataType;
            checkElement(type, split[i], col->readValue(j));
        }
    }
}

This is the message that I print out when the test fails:

size: 22 elements: SyBCmYinMTiPNZgZCvHMxeaAEOARqISVnJ,HatK,KDhBZydMaXxzkaiKntkBgT,eIFuxNmgDDCMlzrMIzSqykP,uPghpGQnlpkkfCuATtF,ECMiUxiKbCGFPQqVBsMVNjrVGIVcSMakrQVcz,egtYmmJvp,zMQnWWCPvpdVBDiUwvrlRlNCwKMZRgxjrti,NGBJFrFQ,xHoWyeSXbpVqUerMqMgmgmlPU,eulTYuXLJopvzGvpXHOkrIarfxa,YCXgSeIXwAptqJkcvWBZSqDeLlEzExQLU,FGiUNyajkoqPSSK,bSTIXLuBcaeovBkOm,diWb,wXauHDvxhiPsWzyUMFSkVVlhrHE,iWAzpbgHltfpSHP,EIdvEQcFAma,mrPGEwFoAofmMlLLKE,zcaeRzOrfqj,IYSvZZmkGlIKIzWFEQpyvPaiqpxrUjkfHg,zLwbAReTVKDaxAzKWkzHctpcoA,
[,,,,,,,,,,,,,,,,,,,,,]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants