error when running cis and analysis

[catherine0202]

hello!
I got an error "Performing block wise correlation calculation Error: upper value must be greater than lower value" when running '--pcatype cis'. But my hic file was complete and had been transformed by 'preprocess.py' without any error.

Here was some part of the log file:
Matrix dimension 14514 X 14514
Performing Z transformation : complete!
Performing block wise correlation calculation : complete!
Matrix dimension 14246 X 14246
Performing Z transformation : complete!
Performing block wise correlation calculation : complete!
chr start end index 1 2
8 chr8 70000 80000 8 -5.427264 -11.900826
14 chr8 130000 140000 14 -26.150188 -5.274410
17 chr8 160000 170000 17 -23.993980 -1.556982
18 chr8 170000 180000 18 -26.141591 -7.992211
20 chr8 190000 200000 20 -24.585561 -6.226422
21 chr8 200000 210000 21 -24.482866 -5.563963
Matrix dimension 13840 X 13840
Performing Z transformation : complete!
Performing block wise correlation calculation : complete!
Matrix dimension 11617 X 11617
Performing Z transformation : complete!
Performing block wise correlation calculation : complete!
chr start end index 1 2
2 chr9 10000 20000 2 5.012175 22.605769
3 chr9 20000 30000 3 -8.487658 26.162420
4 chr9 30000 40000 4 3.589291 26.932818
5 chr9 40000 50000 5 2.808922 9.428807
6 chr9 50000 60000 6 -7.670276 14.440521
7 chr9 60000 70000 7 -6.636064 14.105107
Matrix dimension 2 X 2
Performing Z transformation : complete!
Performing block wise correlation calculation Error: upper value must be greater than lower value
Execution halted.

!!!Then I used the file produced by the above step to run '--pcatype analyze' and also got this error:
Running intra sample differential calls for nfcp2_dchic zk_dchic samples
Error in pcanalyze(data, dirovwt, diffdir, rzscore, szscore, fdr_thr, :
object 'chrom' not found
Execution halted

I installed all required softwares and packages in a slurm-based system, is these errors caused by my input files or the system?

[ay-lab]

Hi,

Any non-standard chromosome (e.g. chrY, chrM and any chromosome with very few interactions) can lead to such error. Please make sure you're providing any such chromosome. '--pcatype analyze' step requires completion of '--pcatype cis' step, so you got the last error. Let me know how it goes.

[catherine0202]

I appreciate your timely and detailed answer, but I still have some confusion. As you say, any non-standard chromosome (e.g. chrY, chrM and any chromosome with very few interactions) may lead to such error, should I remove all of files or information associated with chrY and chrM? and how could I achieve this? Here was some part of my file list after '--pcatype cis' step(files associated with chr1-chr22 had the same format of these results and were omitted.) -rw-r--r-- 1 lhl_zhulin lhl 387K Jun 10 23:11 chr9.bed -rw-r--r-- 1 lhl_zhulin lhl 1.1G Jun 11 02:29 chr9.cmat.txt -rw-r--r-- 1 lhl_zhulin lhl 494K Jun 10 23:11 chr9.distparam -rw-r--r-- 1 lhl_zhulin lhl 721K Jun 11 02:29 chr9.pc.txt -rw-r--r-- 1 lhl_zhulin lhl 1.1G Jun 11 02:26 chr9.precmat.txt -rw-r--r-- 1 lhl_zhulin lhl 348K Jun 11 02:29 chr9.svd.rds -rw-r--r-- 1 lhl_zhulin lhl 299M Jun 10 23:11 chr9.txt -rw-r--r-- 1 lhl_zhulin lhl   54 Jun 10 23:10 chrM.bed -rw-r--r-- 1 lhl_zhulin lhl   75 Jun 10 23:10 chrM.distparam -rw-r--r-- 1 lhl_zhulin lhl  161 Jun 10 23:10 chrM.txt -rw-r--r-- 1 lhl_zhulin lhl 440K Jun 10 23:11 chrX.bed -rw-r--r-- 1 lhl_zhulin lhl 552K Jun 10 23:11 chrX.distparam -rw-r--r-- 1 lhl_zhulin lhl 349M Jun 10 23:11 chrX.txt -rw-r--r-- 1 lhl_zhulin lhl 154K Jun 10 23:11 chrY.bed -rw-r--r-- 1 lhl_zhulin lhl 1.5K Jun 10 23:11 chrY.distparam -rw-r--r-- 1 lhl_zhulin lhl 6.3K Jun 10 23:11 chrY.txt And my log file of '--pcatype cis' step was attached to this email, which contains some information of chrY and chrM. Thank you very much for your answer! and I look forward to receiving your reply.     ------------------ Original ------------------ From: ***@***.***>; Date:  Thu, Jun 15, 2023 12:56 PM To: ***@***.***>; Cc: ***@***.***>; ***@***.***>; Subject:  Re: [ay-lab/dcHiC] error when running cis and analysis (Issue #69)   Hi, Any non-standard chromosome (e.g. chrY, chrM and any chromosome with very few interactions) can lead to such error. Please make sure you're providing any such chromosome. '--pcatype analyze' step requires completion of '--pcatype cis' step, so you got the last error. Let me know how it goes. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ay-lab]

Yes, please delete chrY, chrM files from this folder, coordinate records from the bed files and give it a try again. Hopefully, this will resolve the issue.

[catherine0202]

These problems have been completely solved and&nbsp; I appreciate your detailed answer very much! However, there are another errors when I run "subcomp" and "fithic.py" after runing "analysis" successfully, here is my scripts: Script: Rscript /share/home/lhl_zhulin/zhulin/dchics/dcHiC/dchicf.r --file input.nfcp2_zk.txt --pcatype subcomp --diffdir nfcp2_dchic_vs_zk_dchic_10Kb Error: Error in `[<-.data.frame`(`*tmp*`, , "state", value = 1:6) : &nbsp; replacement has 6 rows, data has 3 Calls: subcompartment -&gt; hmmsegment -&gt; [<- -&gt; [<-.data.frame In addition: There were 16 warnings (use warnings() to see them) Execution halted Script: Rscript /share/home/lhl_zhulin/zhulin/dchics/dcHiC/dchicf.r --file input.nfcp2_zk.txt --pcatype fithic --dirovwt T --diffdir nfcp2_dchic_vs_zk_dchic_10Kb --maxd 10e6 --fithicpath '/share/home/lhl_zhulin/miniconda3/envs/dchic2/lib/python3.9/site-packages/fithic/fithic.py' --pythonpath '/share/home/lhl_zhulin/miniconda3/envs/dchic2/lib/python3.9/python' Error:&nbsp; Fithic requires a bias file. Please check the link for more details https://github.com/ay-lab/fithic Please generate the bias files for each sample provided in the input.txt file Create an additional folder 'biases' under current path and dump all the *.biases.gz files inside it Rerun the step again Error in FUN(X[[i]], ...) : Exit! Calls: fithicformat -&gt; lapply -&gt; FUN Execution halted input.txt nfcp2_dchic_R1_10Kb.matrix&nbsp; &nbsp; &nbsp; nfcp2_dchic_R1_10Kb_abs.bed&nbsp; &nbsp; &nbsp;nfcp2_dchic_R1_10Kb&nbsp; &nbsp; &nbsp;nfcp2_dchic zk_dchic_1_10Kb.matrix&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;zk_dchic_1_10Kb_abs.bed&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;zk_dchic_1_10Kb&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; zk_dchic And full log file was attached to this email. Thanks again! I'm looking forward to your reply.

[ay-lab]

Hi,

To run the fithic part, you first need to have the bias file. Please check this on how to generate the bias files -
https://github.com/ay-lab/fithic#optional-arguments
ay-lab/fithic#53 (comment)

(If you already processed this data using HiCPro then you should have the ice normalized bias values with the "iced" folder)

Please generate the bias files for each sample provided in the input.txt file. Then create an additional folder, 'biases' under the current path and dump all the *.biases.gz files inside it. Rerun the step again.

Regarding the subcompartment error, do you have this file "fdr_result/differential.intra_sample_group.pcQnm.bedGraph"?
Can you share it? It seems, the default subcompartment number (= 6) is creating the issue. You can change this using "--subnum" option (and set it to a lower value like 4???). I hope this resolves the issue.