Datasets in Apache Spark

Datasets in Apache Spark

DatasetTweet ds sc.read.jsonpath.asEncoders.beanTweet.class; ds.show; JavaRDDTweet dstry ds.toJavaRDD; System.out.printlndstry.first.getClass; Caused by: java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File generated.java, Line 50, Column 16: failed to compile: org.codehaus.commons.compiler.CompileException: File generated.java, Line 50, Column 16: No applicable constructormethod found for actual parameters org.apache.spark.unsafe.types.UTF8String; candidates are: public void sparkSQL.Tweet.setIdlong at org.spark_project.guava.util.concurrent.AbstractFutureSync.getValueAbstractFuture.java:306 at org.spark_project.guava.util.concurrent.AbstractFutureSync.getAbstractFuture.java:293 at org.spark_project.guava.util.concurrent.AbstractFuture.getAbstractFuture.java:116 at org.spark_project.guava.util.concurrent.Uninterruptibles.getUninterruptiblyUninterruptibles.java:135 at org.spark_project.guava.cache.LocalCacheSegment.getAndRecordStatsLocalCache.java:2410 at org.spark_project.guava.cache.LocalCacheSegment.loadSyncLocalCache.java:2380 at org.spark_project.guava.cache.LocalCacheSegment.lockedGetOrLoadLocalCache.java:2342 at org.spark_project.guava.cache.LocalCacheSegment.getLocalCache.java:2257 at org.spark_project.guava.cache.LocalCache.getLocalCache.java:4000 at org.spark_project.guava.cache.LocalCache.getOrLoadLocalCache.java:4004 at org.spark_project.guava.cache.LocalCacheLocalLoadingCache.getLocalCache.java:4874 at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.compileCodeGenerator.scala:1369 at org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection.createGenerateSafeProjection.scala:197 at org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection.createGenerateSafeProjection.scala:36 at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generateCodeGenerator.scala:1325 at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generateCodeGenerator.scala:1322 at org.apache.spark.sql.execution.DeserializeToObjectExecanonfun2.applyobjects.scala:90 at org.apache.spark.sql.execution.DeserializeToObjectExecanonfun2.applyobjects.scala:89 at org.apache.spark.rdd.RDDanonfunmapPartitionsWithIndexInternal1anonfunapply24.applyRDD.scala:818 at org.apache.spark.rdd.RDDanonfunmapPartitionsWithIndexInternal1anonfunapply24.applyRDD.scala:818 at org.apache.spark.rdd.MapPartitionsRDD.computeMapPartitionsRDD.scala:38 at org.apache.spark.rdd.RDD.computeOrReadCheckpointRDD.scala:324 at org.apache.spark.rdd.RDD.iteratorRDD.scala:288 at org.apache.spark.rdd.MapPartitionsRDD.computeMapPartitionsRDD.scala:38 at org.apache.spark.rdd.RDD.computeOrReadCheckpointRDD.scala:324 at org.apache.spark.rdd.RDD.iteratorRDD.scala:288 at org.apache.spark.scheduler.ResultTask.runTaskResultTask.scala:87 at org.apache.spark.scheduler.Task.runTask.scala:109 at org.apache.spark.executor.ExecutorTaskRunner.runExecutor.scala:345 at java.util.concurrent.ThreadPoolExecutor.runWorkerThreadPoolExecutor.java:1149 at java.util.concurrent.ThreadPoolExecutorWorker.runThreadPoolExecutor.java:624 at java.lang.Thread.runThread.java:748 Caused by: org.codehaus.commons.compiler.CompileException: File generated.java, Line 50, Column 16: failed to compile: org.codehaus.commons.compiler.CompileException: File generated.java, Line 50, Column 16: No applicable constructormethod found for actual parameters org.apache.spark.unsafe.types.UTF8String; candidates are: public void sparkSQL.Tweet.setIdlong at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.orgapachesparksqlcatalystexpressionscodegenCodeGeneratordoCompileCodeGenerator.scala:1435 at org.apache.spark.sql.catalyst.expressions.codegen.CodeGeneratoranon1.loadCodeGenerator.scala:1497 at org.apache.spark.sql.catalyst.expressions.codegen.CodeGeneratoranon1.loadCodeGenerator.scala:1494 at org.spark_project.guava.cache.LocalCacheLoadingValueReference.loadFutureLocalCache.java:3599 at org.spark_project.guava.cache.LocalCacheSegment.loadSyncLocalCache.java:2379 When I looked closely the only thing I raised my doubts is this: No applicable constructormethod found for actual parameters org.apache.spark.unsafe.types.UTF8String; candidates are: public void sparkSQL.Tweet.setIdlong

It gives you an error because of type mismatch: Tweet class defines id field as Long. Your data has id as String. You have to either cast input or adjust class definition.

Комментарии

Отправить комментарий

Популярные сообщения из этого блога

Skipping acquire of configured file 'contrib/binary-i386/Packages' as repository … doesn't support architecture 'i386'

Skipping acquire of configured file 'contrib/binary-i386/Packages' as repository … doesn't support architecture 'i386' I installed Ubuntu 18.04 and then while installing Virtualbox i get the i386 error message after I did: sudo apt-get update Get:1 http:nl.archive.ubuntu.comubuntu bionic InRelease 242 kB Hit:2 http:security.ubuntu.comubuntu bionic-security InRelease Hit:3 http:ppa.launchpad.netdanielrichter2007grub-customizerubuntu bionic InRelease Hit:4 http:nl.archive.ubuntu.comubuntu bionic-updates InRelease Hit:5 http:nl.archive.ubuntu.comubuntu bionic-backports InRelease Get:6 http:download.virtualbox.orgvirtualboxdebian bionic InRelease 4429 B Get:7 http:download.virtualbox.orgvirtualboxdebian bioniccontrib amd64 Packages 1426 B Fetched 248 kB in 1s 209 kBs Reading package lists... Done N: Skipping acquire of configured file contribbinary-i386Packages as repository http:download.virtualbox.orgvirtualboxdebian bionic InRelease doesnt support architecture i386 ...
Далее...

Connection string for MariaDB using ODBC

Connection string for MariaDB using ODBC What is the connection string for using MariaDB with ODBC Im guessing its something like DRIVERMariaDB ODBC 5.1 Driver; SERVERLOCALHOST; PORT3306; DATABASEDatabaseName; UIDroot; PASSWORDpassword;OPTION3; Connection strings seems to show strings for MySQL but not MariaDB. MariaDB seems to have a lot of connection strings for Access, SQL Lite, excel, etc ... What have I missed Although this is late answer, but MariaDB have that documented here SQLWCHAR ConnStr LDriverMariaDB ODBC 2.0 Driver;Serverlocalhost;UIDodbc_user;PWDodbc_pw;DBodbc_test;Port3306;
Далее...

Celery like system based on django channels

Celery like system based on django channels According to the docs of django channels: http:channels.readthedocs.ioenlatestconcepts.htmlhighlightcelerynext-steps One thing channels do not do, however, is guarantee delivery. If you need certainty that tasks will complete, use a system designed for this with retries and persistence e.g. Celery, or alternatively make a management command that checks for completion and re-submits a message to the channel if nothing is completed rolling your own retry logic, essentially. I worked with celery some years ago. Yes, it is great, it is big, it is too much. I would like to avoid it. I am searching a simple tasks queue with guarantee delivery and a re-submission on failure. An other alternative would be python-rq. But this has other disadvantages like using pickle data format instead of json. A friend gave me an advice: https:github.comfurious-lukedjango-cq Looks good. From the docs An attempt to implement a distributed task queue ...
Далее...