Flume Spooling Directory Source 采集NullPointerException
采用flume的Spooling Directory Source,突然遇到下面的错误
(org.apache.flume.source.SpoolDirectoryExtSource2$SpoolDirectoryRunnable.run:277) - FATAL: Spool Directory source source1: { spoolDir: /home/work/local/log }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
java.lang.NullPointerException
at org.apache.avro.io.ResolvingDecoder.readLong(ResolvingDecoder.java:159)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:157)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
at org.apache.flume.serialization.DurablePositionTracker.initReader(DurablePositionTracker.java:171)
at org.apache.flume.serialization.DurablePositionTracker.<init>(DurablePositionTracker.java:158)
at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:76)
at org.apache.flume.client.avro.ReliableSpoolingFileEventExtReader2.openFile(ReliableSpoolingFileEventExtReader2.java:561)
at org.apache.flume.client.avro.ReliableSpoolingFileEventExtReader2.getNextFile(ReliableSpoolingFileEventExtReader2.java:511)
at org.apache.flume.client.avro.ReliableSpoolingFileEventExtReader2.readEvents(ReliableSpoolingFileEventExtReader2.java:264)
at org.apache.flume.source.SpoolDirectoryExtSource2$SpoolDirectoryRunnable.run(SpoolDirectoryExtSource2.java:252)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
重启flume 后发现问题依旧。试验把flume 采集的日志放到新目录,重启flume采集新目录发现没问题了。看来是旧目录/home/work/local/log 有文件变动造成的。
例如 有.flumespool隐藏目录,该目录下有文件.flumespool-main.meta隐藏文件,用来记录flume读取文件的位置,发现该记录停止在flume出问题的时间。再查看其它正常运行的机器的相同路径及文件,并没有发现该文件,于是将该文件移到其它目录下,重启flume,此时发现flume成功运行!