FORUMmegahit error
spongemicrobiome asked 3 months ago



Hi There,
I am trying to use the program megahit for assembly of a metagenome.  I have the program loaded but it fails after about ~5mins of running.  I am calling the program in a script as follows:

module load gcc/7.3.0 cuda/10.0.130 megahit/1.2.7
megahit -1 /home/path_to_forward_reads_/R1_P_trimmed.fq.gz -2 /home/path_to_reverse_reads_/R2_P_trimmed.fq.gz -m 0.5 -t 16 –k-min 27 –k-max 147 –k-step 10 -o /home/path_to_output_dir/assembly/

I thought maybe it was a memory issue so i capped that with “-m 0.5“, and only using “-t 16” . This still did not fix the issue. 
I get a very large file (size = 3368841216) called “core.4477” which i have never seen produced from megahit before and if less it its pages of:
F+^@^@^@<B0>
!F+^@^@<98>^A^@^@^@^@^@^@^@<F0>
!F+^@^@^@^P^K!F+^@^@^@^@^@^@^@^@^@^@^@^P^K!F+^@^@^@^P+!F+^@^@^B^@^@^@^@^@^@^@^@
^P+!F+^@^@^@ +!F+^@^@^B^@^@^@^@^@^@^@^@ +!F+^@^@^@0+!F+^@^@^C^@^@^@^@^@^@^@/cvmf
s/soft.computecanada.ca/easybuild/software/2017/avx2/CUDA/gcc7.3/cuda10.0/megahi
t/1.2.7/bin/megahit_core^@/cvmfs/soft.computecanada.ca/easybuild/software/2017/a
vx2/CUDA/gcc7.

When i read the slurm there is not much info as to why its failing:

2019-09-17 10:25:47 – Convert reads to binary library
2019-09-17 10:27:48 – INFO  sequence/io/sequence_lib.cpp  :   77 – Lib 0 (/home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/trimreads/SEEP_BRIDGE_DI_R1_P_trimmed.fq.gz,/home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/trimreads/SEEP_BRIDGE_DI_R2_P_trimmed.fq.gz): pe, 72595302 reads, 151 max length
2019-09-17 10:27:48 – INFO  utils/utils.h                 :  152 – Real: 120.5771       user: 91.9506   sys: 21.9402    maxrss: 247472
2019-09-17 10:27:48 – Start assembly. Number of CPU threads 16
2019-09-17 10:27:48 – k list: 27,37,47,57,67,77,87,97,107,117,127,137,147
2019-09-17 10:27:48 – Memory used: 67537805312
2019-09-17 10:27:48 – Extract solid (k+1)-mers for k = 27
2019-09-17 10:33:01 – Build graph for k = 27
2019-09-17 10:38:06 – Assemble contigs from SdBG for k = 27
2019-09-17 10:46:30 – Error occurs, please refer to /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/log for detail
2019-09-17 10:46:30 – Command: /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/CUDA/gcc7.3/cuda10.0/megahit/1.2.7/bin/megahit_core assemble -s /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/tmp/k27/27 -o /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/intermediate_contigs/k27 -t 16 –min_standalone 300 –prune_level 2 –merge_len 20 –merge_similar 0.95 –cleaning_rounds 5 –disconnect_ratio 0.1 –low_local_ratio 0.2 –cleaning_rounds 5 –min_depth 2 –bubble_level 2 –max_tip_len -1 –careful_bubble; Exit code -6

Here is the output of the log file:
2019-09-17 10:25:47 – MEGAHIT v1.2.7
2019-09-17 10:25:47 – Using megahit_core with POPCNT and BMI2 support
2019-09-17 10:25:47 – Convert reads to binary library
2019-09-17 10:25:47 – command /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/CUDA/gcc7.3/cuda10.0/megahit/1.2.7/bin/megahit_core buildlib /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/tmp/reads.lib /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/tmp/reads.lib
2019-09-17 10:27:48 – INFO sequence/io/sequence_lib.cpp : 77 – Lib 0 (/home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/trimreads/SEEP_BRIDGE_DI_R1_P_trimmed.fq.gz,/home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/trimreads/SEEP_BRIDGE_DI_R2_P_trimmed.fq.gz): pe, 72595302 reads, 151 max length
2019-09-17 10:27:48 – INFO utils/utils.h : 152 – Real: 120.5771 user: 91.9506 sys: 21.9402 maxrss: 247472
2019-09-17 10:27:48 – Start assembly. Number of CPU threads 16
2019-09-17 10:27:48 – k list: 27,37,47,57,67,77,87,97,107,117,127,137,147
2019-09-17 10:27:48 – Memory used: 67537805312
2019-09-17 10:27:48 – Extract solid (k+1)-mers for k = 27
2019-09-17 10:27:48 – command /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/CUDA/gcc7.3/cuda10.0/megahit/1.2.7/bin/megahit_core count -k 27 -m 2 –host_mem 67537805312.0 –mem_flag 1 –output_prefix /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/tmp/k27/27 –num_cpu_threads 16 –read_lib_file /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/tmp/reads.lib
2019-09-17 10:27:48 – INFO sorting/base_engine.cpp : 148 – Preparing data…
2019-09-17 10:27:48 – INFO sequence/io/sequence_lib.cpp : 115 – Before reading, sizeof seq_package: 3215614856
2019-09-17 10:27:57 – INFO sequence/io/sequence_lib.cpp : 117 – After reading, sizeof seq_package: 3215614856
2019-09-17 10:27:57 – INFO sorting/kmer_counter.cpp : 76 – 72595302 reads, 151 max read length
2019-09-17 10:27:57 – INFO sorting/kmer_counter.cpp : 83 – 2 words per substring, 3 words per edge
2019-09-17 10:27:57 – INFO sorting/base_engine.cpp : 153 – Preparing data… Done. Time elapsed: 9.6948
2019-09-17 10:27:57 – INFO sorting/base_engine.cpp : 156 – Preparing partitions and calculating bucket sizes…
2019-09-17 10:28:08 – INFO sorting/base_engine.cpp : 80 – Minimum memory required: 4202744292 bytes
2019-09-17 10:28:11 – INFO sorting/base_engine.cpp : 138 – Lv1 items: 1143937871, Lv2 items: 6283824
2019-09-17 10:28:11 – INFO sorting/base_engine.cpp : 140 – Memory of derived class: 3887105048, Memory for Lv1+Lv2: 4676292668
2019-09-17 10:28:11 – INFO sorting/base_engine.cpp : 168 – Preparing partitions and calculating bucket sizes… Done. Time elapsed: 13.4045
2019-09-17 10:28:11 – INFO sorting/base_engine.cpp : 172 – Start main loop…
2019-09-17 10:28:11 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 0 to 1046
2019-09-17 10:28:24 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 12.8581
2019-09-17 10:28:40 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 16.9110
2019-09-17 10:28:40 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 1046 to 3148
2019-09-17 10:28:54 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 13.6388
2019-09-17 10:29:14 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 19.5220
2019-09-17 10:29:14 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 3148 to 6238
2019-09-17 10:29:28 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 14.1731
2019-09-17 10:29:49 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 21.2304
2019-09-17 10:29:49 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 6238 to 10512
2019-09-17 10:30:04 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 14.7469
2019-09-17 10:30:26 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 22.0881
2019-09-17 10:30:26 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 10512 to 16434
2019-09-17 10:30:41 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 15.2846
2019-09-17 10:31:04 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 23.1813
2019-09-17 10:31:04 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 16434 to 24991
2019-09-17 10:31:20 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 15.6392
2019-09-17 10:31:44 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 24.0282
2019-09-17 10:31:44 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 24991 to 39394
2019-09-17 10:32:00 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 16.1445
2019-09-17 10:32:26 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 25.6580
2019-09-17 10:32:26 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 39394 to 65536
2019-09-17 10:32:40 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 14.1519
2019-09-17 10:32:56 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 16.3075
2019-09-17 10:32:56 – INFO sorting/base_engine.cpp : 204 – Main loop done. Time elapsed: 285.5726
2019-09-17 10:32:56 – INFO sorting/base_engine.cpp : 207 – Postprocessing…
2019-09-17 10:33:00 – INFO sorting/kmer_counter.cpp : 406 – Total number of candidate reads: 4537891 (16253539)
2019-09-17 10:33:00 – INFO sorting/kmer_counter.cpp : 408 – Total number of solid edges: 948756888
2019-09-17 10:33:00 – INFO sorting/base_engine.cpp : 210 – Postprocess done. Time elapsed: 4.0569
2019-09-17 10:33:01 – INFO utils/utils.h : 152 – Real: 313.4382 user: 4613.5264 sys: 106.0754 maxrss: 8399168
2019-09-17 10:33:01 – Build graph for k = 27
2019-09-17 10:33:01 – command /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/CUDA/gcc7.3/cuda10.0/megahit/1.2.7/bin/megahit_core seq2sdbg –host_mem 67537805312.0 –mem_flag 1 –output_prefix /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/tmp/k27/27 –num_cpu_threads 16 -k 27 –kmer_from 0 –input_prefix /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/tmp/k27/27 –need_mercy
2019-09-17 10:33:01 – INFO sorting/base_engine.cpp : 148 – Preparing data…
2019-09-17 10:33:01 – INFO sorting/seq_to_sdbg.cpp : 369 – Number edges: 948756888
2019-09-17 10:33:01 – INFO sorting/seq_to_sdbg.cpp : 415 – Bases to reserve: 33206491080, number contigs: 0, number multiplicity: 1185946110
2019-09-17 10:33:01 – INFO sorting/seq_to_sdbg.cpp : 422 – Before reading, sizeof seq_package: 8301622780, multiplicity vector: 1185946110
2019-09-17 10:33:42 – INFO sorting/seq_to_sdbg.cpp : 428 – Read 948756888 edges.
2019-09-17 10:33:42 – INFO sorting/seq_to_sdbg.cpp : 433 – After reading, sizeof seq_package: 8301622780/948756888/26565192864, multiplicity vector: 948756888/1185946110
2019-09-17 10:33:42 – INFO sorting/seq_to_sdbg.cpp : 440 – Adding mercy edges…
2019-09-17 10:34:04 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:05 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:06 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:08 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:09 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:10 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:11 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:13 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:15 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:16 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:17 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:18 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:20 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:21 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:22 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:23 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:24 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:26 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:27 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:28 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:29 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:31 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:32 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:33 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:34 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:36 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:37 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:38 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:39 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:41 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:42 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:43 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:44 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:46 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:47 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:48 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:49 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:51 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:52 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:53 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:54 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:56 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:57 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:58 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:34:59 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:01 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:02 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:03 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:04 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:05 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:07 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:08 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:09 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:10 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:12 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:13 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:14 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:15 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:17 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:18 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:19 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:20 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:22 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:23 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:24 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:25 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:27 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:28 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:29 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 65536 reads to search for mercy k-mers
2019-09-17 10:35:30 – INFO sorting/seq_to_sdbg.cpp : 192 – Read 15907 reads to search for mercy k-mers
2019-09-17 10:35:31 – INFO sorting/seq_to_sdbg.cpp : 356 – Number of reads: 4537891, Number of mercy edges: 137033696
2019-09-17 10:35:31 – INFO sorting/seq_to_sdbg.cpp : 444 – Done. Time elapsed: 108.5788
2019-09-17 10:35:31 – INFO sorting/seq_to_sdbg.cpp : 449 – After adding mercy, sizeof seq_package: 8301622780/1085790584/30402136352, multiplicity vector: 1085790584/1185946110
2019-09-17 10:35:31 – INFO sorting/seq_to_sdbg.cpp : 508 – Finally, sizeof seq_package: 8301622780/1085790584/30402136352, multiplicity vector: 1085790584/1185946110
2019-09-17 10:35:31 – INFO sorting/base_engine.cpp : 153 – Preparing data… Done. Time elapsed: 149.6824
2019-09-17 10:35:31 – INFO sorting/base_engine.cpp : 156 – Preparing partitions and calculating bucket sizes…
2019-09-17 10:35:40 – INFO sorting/base_engine.cpp : 80 – Minimum memory required: 11139101008 bytes
2019-09-17 10:35:43 – INFO sorting/base_engine.cpp : 138 – Lv1 items: 868632467, Lv2 items: 4771536
2019-09-17 10:35:43 – INFO sorting/base_engine.cpp : 140 – Memory of derived class: 10911031720, Memory for Lv1+Lv2: 3531788300
2019-09-17 10:35:43 – INFO sorting/base_engine.cpp : 168 – Preparing partitions and calculating bucket sizes… Done. Time elapsed: 11.6378
2019-09-17 10:35:43 – INFO sorting/base_engine.cpp : 172 – Start main loop…
2019-09-17 10:35:43 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 0 to 1575
2019-09-17 10:35:53 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 10.5633
2019-09-17 10:36:03 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 10.3918
2019-09-17 10:36:03 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 1575 to 4689
2019-09-17 10:36:10 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 6.6244
2019-09-17 10:36:21 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 10.8340
2019-09-17 10:36:21 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 4689 to 9223
2019-09-17 10:36:28 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 6.7369
2019-09-17 10:36:39 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 10.9330
2019-09-17 10:36:39 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 9223 to 15277
2019-09-17 10:36:46 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 6.9538
2019-09-17 10:36:57 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 10.9638
2019-09-17 10:36:57 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 15277 to 23201
2019-09-17 10:37:04 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 7.4609
2019-09-17 10:37:15 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 11.0484
2019-09-17 10:37:15 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 23201 to 33750
2019-09-17 10:37:23 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 7.4965
2019-09-17 10:37:34 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 11.1667
2019-09-17 10:37:34 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 33750 to 49499
2019-09-17 10:37:42 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 7.7935
2019-09-17 10:37:53 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 11.2160
2019-09-17 10:37:53 – INFO sorting/base_engine.cpp : 186 – Lv1 scanning from bucket 49499 to 65536
2019-09-17 10:37:59 – INFO sorting/base_engine.cpp : 193 – Lv1 scanning done. Large diff: 0. Time elapsed: 6.2833
2019-09-17 10:38:05 – INFO sorting/base_engine.cpp : 199 – Lv1 fetching & sorting done. Time elapsed: 5.8518
2019-09-17 10:38:05 – INFO sorting/base_engine.cpp : 204 – Main loop done. Time elapsed: 142.3261
2019-09-17 10:38:05 – INFO sorting/base_engine.cpp : 207 – Postprocessing…
2019-09-17 10:38:05 – INFO sorting/seq_to_sdbg.cpp : 793 – Number of $ A C G T A- C- G- T-:
2019-09-17 10:38:05 – INFO sorting/seq_to_sdbg.cpp : 794 – 14742369 455415904 633657824 635116617 455152761 1084447 2405351 2421353 1067631
2019-09-17 10:38:05 – INFO sorting/seq_to_sdbg.cpp : 800 – Total number of edges: 2201064257
2019-09-17 10:38:05 – INFO sorting/seq_to_sdbg.cpp : 801 – Total number of ONEs: 2179343106
2019-09-17 10:38:05 – INFO sorting/seq_to_sdbg.cpp : 803 – Total number of $v edges: 14742369
2019-09-17 10:38:05 – INFO sorting/base_engine.cpp : 210 – Postprocess done. Time elapsed: 0.0642
2019-09-17 10:38:06 – INFO utils/utils.h : 152 – Real: 304.9620 user: 3416.7373 sys: 71.1459 maxrss: 13081104
2019-09-17 10:38:06 – Assemble contigs from SdBG for k = 27
2019-09-17 10:38:06 – command /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/CUDA/gcc7.3/cuda10.0/megahit/1.2.7/bin/megahit_core assemble -s /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/tmp/k27/27 -o /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/intermediate_contigs/k27 -t 16 –min_standalone 300 –prune_level 2 –merge_len 20 –merge_similar 0.95 –cleaning_rounds 5 –disconnect_ratio 0.1 –low_local_ratio 0.2 –cleaning_rounds 5 –min_depth 2 –bubble_level 2 –max_tip_len -1 –careful_bubble
2019-09-17 10:38:53 – INFO main_assemble.cpp : 129 – Loading succinct de Bruijn graph: /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/tmp/k27/27Done. Time elapsed: 47.008568
2019-09-17 10:38:53 – INFO main_assemble.cpp : 133 – Number of Edges: 2201064257; K value: 27
2019-09-17 10:38:53 – INFO main_assemble.cpp : 140 – Number of CPU threads: 16
2019-09-17 10:39:20 – INFO assembly/sdbg_pruning.cpp : 160 – Removing tips with length less than 2; Accumulated tips removed: 368650; time elapsed: 2.6854
2019-09-17 10:39:25 – INFO assembly/sdbg_pruning.cpp : 160 – Removing tips with length less than 4; Accumulated tips removed: 1100561; time elapsed: 4.6705
2019-09-17 10:39:35 – INFO assembly/sdbg_pruning.cpp : 160 – Removing tips with length less than 8; Accumulated tips removed: 2089344; time elapsed: 10.1763
2019-09-17 10:39:57 – INFO assembly/sdbg_pruning.cpp : 160 – Removing tips with length less than 16; Accumulated tips removed: 3621395; time elapsed: 21.7634
2019-09-17 10:40:37 – INFO assembly/sdbg_pruning.cpp : 160 – Removing tips with length less than 32; Accumulated tips removed: 5806553; time elapsed: 39.9274
2019-09-17 10:41:33 – INFO assembly/sdbg_pruning.cpp : 169 – Removing tips with length less than 54; Accumulated tips removed: 7829599; time elapsed: 56.6510
2019-09-17 10:41:33 – INFO main_assemble.cpp : 158 – Tips removal done! Time elapsed(sec): 160.040
2019-09-17 10:45:02 – INFO assembly/unitig_graph.cpp : 84 – Graph size without loops: 12073549, palindrome: 1500
2019-09-17 10:45:12 – INFO main_assemble.cpp : 167 – unitig graph size: 12085257, time for building: 218.576
2019-09-17 10:45:32 – INFO assembly/contig_stat.h : 40 – Max: 23761, Min: 28, N50: 147, number contigs: 12085257, number isolated: 3153332, number looped: 11708, total size: 1332397452,
2019-09-17 10:45:32 – INFO main_assemble.cpp : 184 – Graph cleaning round 1
2019-09-17 10:45:47 – INFO main_assemble.cpp : 201 – Number of bubbles removed: 727321, Time elapsed(sec): 14.906
2019-09-17 10:45:56 – INFO main_assemble.cpp : 211 – Number of complex bubbles removed: 219193, Time elapsed(sec): 9.180119
2019-09-17 10:45:59 – INFO main_assemble.cpp : 222 – Number unitigs disconnected: 591974, time: 2.664
2019-09-17 10:46:00 – INFO main_assemble.cpp : 246 – Unitigs removed in excessive pruning: 808, time: 1.403
2019-09-17 10:46:00 – INFO main_assemble.cpp : 184 – Graph cleaning round 2
2019-09-17 10:46:07 – INFO main_assemble.cpp : 192 – Tips removed: 127963, time: 6.796
2019-09-17 10:46:09 – megahit_core: /dev/shm/ebuser/avx2/MEGAHIT/1.2.7/gmklc-2018.3.100/megahit-1.2.7/src/assembly/unitig_graph.cpp:315: void UnitigGraph::Refresh(bool): Assertion `!(next_adapter.GetFlag() & kDeleted)’ failed.
2019-09-17 10:46:30 – Error occurs, please refer to /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/log for detail
2019-09-17 10:46:30 – Command: /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/CUDA/gcc7.3/cuda10.0/megahit/1.2.7/bin/megahit_core assemble -s /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/tmp/k27/27 -o /home/sacrowe/projects/def-sacrowe/sacrowe/HK_Fe_Seep/assembly/Seep_bridge/intermediate_contigs/k27 -t 16 –min_standalone 300 –prune_level 2 –merge_len 20 –merge_similar 0.95 –cleaning_rounds 5 –disconnect_ratio 0.1 –low_local_ratio 0.2 –cleaning_rounds 5 –min_depth 2 –bubble_level 2 –max_tip_len -1 –careful_bubble; Exit code -6

Any ideas?  Or should i send this to the creators of megahit? 

Thanks, rachel 

2 Answers
jrosner Staff answered 3 months ago



Hi Rachel, 
The file “core.4477” is called a core dump, and it’s generated by the system rather than the software.  It happens when a process terminates unexpectedly and so it just dumps everything from it’s memory space into a file.  Most likely this is caused by a problem with the code and so, I think contacting the megahit authors could be your best bet.
I did find this online:
https://github.com/voutcn/megahit/issues/114
looks like the same issue.  there are a couple of suggestions in that thread, but doesn’t look like they reached a solution.
sorry i can’t provide a solution, but hope this helps
 

spongemicrobiome replied 3 months ago

ok i’ll contact the developers. Thanks!

spongemicrobiome answered 3 months ago



Hi There,
Thought i’d pass on the info from the megahit people.  Is this something i can fix myself?
 
Short answer:
Please just compile with -DCMAKE_BUILD_TYPE=Release to ignore the assertion.
The assertion is incorrect. It is okay to remove it. I am yet to come out with a toy example and add it as a test case.
The released binary was compiled with -DCMAKE_BUILD_TYPE=Release which will ignore all assertions. So users who use pre-compiled binaries are not affected.
@spongebob22 encountered the same issue. While it failed on the server, the assembly was successful on a local machine. I guess the binaries on the local machine was compiled with the release flag while that on the server was not.
 
Thanks, 
 
Rachel 

spongemicrobiome replied 3 months ago

also is this something i can do when i do “load module”?

jrosner Staff replied 3 months ago

thanks!
and wrt the ‘load module’, the answer is no. loading the module makes it available within your environment, but the flag option needs to be invoked during installation (i.e. when the code is compiled).
So, there are 2 choices, you can install this yourself in your home directory, following the steps and using the suggested flag, or you can send an email to support@computecanada.ca and request that they recompile the system version of the software using that flag. if you go this route (which is probably best), you should include the message you received from the developers.