Posts from the Organizers
- Challenge Closed. Thank You To Those Who Participated! September 15, 2018
- Paper review comments August 2, 2018
- Paper Reviews July 29, 2018
- Questions for the Challenge March 18, 2018
Recent Comments
- Zhaohan Xiong on Questions for the Challenge
- Nick on Questions for the Challenge
- Zhaohan Xiong on Discussion Forum
- Vesal Sulaiman on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Xin Yang on Discussion Forum
- Jichao Zhao on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Sulaiman Vesal on Discussion Forum
- Jichao Zhao on Discussion Forum
- Xin Yang on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Xin Yang on Discussion Forum
- Zhaohan Xiong on Paper review comments
- Sulaiman Vesal on Paper review comments
- Zhaohan Xiong on Discussion Forum
- Sandy on Discussion Forum
- Zhaohan Xiong on Paper Reviews
- Sulaiman Vesal on Paper Reviews
- Zhaohan Xiong on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Xin Yang on Discussion Forum
- Jianqiang Ma on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- YashuLiu on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Xin Yang on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Chen Chen on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Sulaiman Vesal on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Xin Yang on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Rashed Karim on Discussion Forum
- Zhaohan Xiong on Questions for the Challenge
- Luyang LUO on Questions for the Challenge
- Zhaohan Xiong on Questions for the Challenge
- Rashed Karim on Questions for the Challenge
- Zhaohan Xiong on Questions for the Challenge
- Luyang LUO on Questions for the Challenge
- Zhaohan Xiong on Discussion Forum
- Xin Yang on Discussion Forum
- Zhaohan Xiong on Questions for the Challenge
- Luyang LUO on Questions for the Challenge
- Zhaohan Xiong on Discussion Forum
- Sandy on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Chen Chen on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Qing Xia on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Sandy on Discussion Forum
- Zhaohan Xiong on Discussion Forum
- Zhaohan Xiong on Questions for the Challenge
- Qing Xia on Discussion Forum
- Shamane Siriwardana on Questions for the Challenge
- Zhaohan Xiong on Discussion Forum
- Sulaiman Vesal on Discussion Forum
- Zhao, Na on Questions for the Challenge
- Zhaohan Xiong on Questions for the Challenge
- Zhao, Na on Questions for the Challenge
Dear Zhaohan Xiong,
We have completed our prediction, but unfortunately, we do not remember the ID that we registered our team for this challenge. Do you have any alternative solution to resolve this issue?
Hi Vesal Sulaiman
As long as you use the same ID for all your submissions and the ID/email you use is something that is easy for us to recognize who you are, then its fine.
Thanks
Zhaohan
Dear Organizers,
Could you kindly provide some comments on the testing results? We can only get the highest Dice score in the result email, rather than the average Dice over all testing samples. We hope to get the average value, since it should be more informative than the highest one.
thanks,
xin
Hi Xin
The value sent in the email is indeed the average DICE over all test samples, not the highest DICE score from any individual test data.
Thanks
Zhaohan.
Dear atrial segmentation challenge participants,
This email is a friendly reminder that our 54 testing data will be released shortly before the starting of the 1 September (UTC) and final competition ends on 14 September (UTC).
For submission of your final results, check the link below:
http://atriaseg2018.cardiacatlas.org/submission/
Important note for final challenge scoring and ranking:
1. The approaches we seek to evaluate in this challenge need to be automatic (no manual modification);
2. We will ask for the final five top-ranked teams to submit their codes to us for further validation in addition to their result submission (by 14 Sept). Thus if you think you are doing well for the testing data, better to submit your final results and codes to us at the same time.
Please note that any researcher/s can participate in the challenge submission/evaluation as well as the bench-marking study following the challenge regardless of whether or not they attend the STACOM conference. However, the monetary prize will only be award based on the best DICE score during the evaluation stage of the challenge and is only available for those who has their paper accepted and presented at the STACOM workshop.
Any further questions, please post online:
http://atriaseg2018.cardiacatlas.org/discussion-forum/
Dear Organizers,
As you have mentioned in submission section: “*It is highly recommended participants use the template submission file as it includes all the rows necessary for prediction.”
However, the template submission file is the old version and It’s only for 25 patients, not 54.
Hi Sulaiman
The updated template will be provided when the test set is released tomorrow.
Thanks.
Zhaohan.
Dear Organizers,
we just received emails from STACOM about the release of all submitted papers. Why the papers full of technical details and results are released before the final competition?
At the beginning of the Challenge, the rule is that “no paper submission, no final testing”. Then, the rule is changed as in the “Paper Review Comments” section, “For those who did not submit a paper, we still encourage you to submit entries for the challenge, as you will be scored on the same leaderboard.”
We can accept this kind of rule change. However, releasing the paper of teams before the final testing is definitely unfair. Those teams who didn’t submit their papers can potentially benefit from those released papers with respect to methods and results, while they don’t need to disclose their methods and results for other teams for reference.
This is definitely unfair for the teams who have prepared for this Challenge for months.
We hope the organizers can explain on this problem.
thanks,
xin
Hi Xin
I have already replied to you via email but I think this will benefit other participants. We do not have control of conference paper publishing so we will address this issue with the conference organizers. Also, the people who did not submit a paper will only be scored, and will not receive the prize regardless of their score. This rule remained the same throughout the challenge and we did not change it.
Thanks.
Zhaohan.
Dear Zhaohan,
thanks for the reply and explanation.
Please receive my apologize for my previous comments.
best,
xin
many thanks for your email and concern, apologize if it is not clear enough since it is really our job to make it fair, open and encourage more people to participate.
THE RULE for our challenge is same and consistent ALWAYs: that is to claim prize and place in the final challenge, one will need to submit the MICCAI paper and present there in STACOM either in poster or oral. However, one misses the paper submission, can still submit their results to us for evaluation, and may have a chance to participate the bench-marking study as an joint journal paper (as well as high performances of the MICCAI participants). However, we do agree we may change the wording a little bit in different stages, but the RULE is same.
When to release the drafted paper is also same and consistent year after year by the MICCAI conference, I am not sure we can stop them for sharing prior to conference, but I will send an email for asking them anyway. Secondly, everyone participating the atrial challenge is sharing their papers at the same time, we will not see any problem here. Since everyone is using different approaches, if someone does copy your approach, we are sure we can use the originally submitted paper to trace it.
On the contrary, to make the challenge more fair, we are doing everything we can to improve it
1) by increasing more testing data to 54 3D LGE-MRI; and
2) asking the final 5 top ranked teams to submit their codes for further validating in addition to their result submission.
Finally, we would advise everyone to focus on improving their approaches instead of worrying something absolutely unnecessary in your final days.
many thanks,
sincerely
Jichao ZHAO
Dear organizers,
can you tell us how many submissions you have received for the challenge? It is interesting to know how many teams actually participate.
Regards,
Sandy
Hi Sandy
There have been over 150 different individuals who have accessed our data. We have had 17 conference paper submissions. With this being said, we are still not sure how many exactly will participate in the final evaluation stage of the challenge. Hope this information helps nevertheless.
Thanks
Zhaohan
Dear Organizers,
as the review process has finished, we want to know, when can we get the detailed comments from reviewers? since it’s only about 1 week before the camera-ready deadline.
thanks,
xin
Hi Xin
We are working towards getting feedback, however, we will still need to discuss with the STACOM organizers to give a final review. Please be patient as there were a lot of submissions, particularly for this challenge.
Thanks
Zhaohan
Hi,
Because the test set is open too late, I would like to know if the organizer can publish the best result of the validationset according to the paper results.Although the reliability of the result is not too high, because of the difference of the validation, it still has certain reference value for us.
Thanks,
Jianqiang Ma
Hi Jianqiang
The reason we are only allowing 2 weeks for participants to predict for the test set is to avoid manual labeling as if we release it months prior, then it would be easier to manually segment the atria, which is not the point of the competition. Unfortunately, it is not possible to publish all the results from the preliminary validation, as the difference in validation strategies results in vastly different scores which might be confusing for some people. However, I can tell you that some teams have achieved roughly 0.90 DICE scores when performing validation on the training sets.
Thanks
Zhaohan
Hi,
I’m wondering why the paper submission website just has submission abstract button. Is the paper submitted when I submit the paper in submission abstract? I hope you can understand me.
Thanks.
Hi Yashu
According to the organizers of STACOM, you can:
1) register the abstract and upload the paper a (browse document PDF) at that time (see attached); or
2) register and later click on the title of the paper, and you should have a green button at the bottom to click for upload of the paper (see snapshot). You can also re-upload a new PDF version which will override the previous one.
Thanks
Zhaohan
Dear Organizers,
apologize to bother you again.
for the full paper submission deadline, there seems to be some inconsistency.
On the Challenge website, the full paper deadline is
“28 June 2018 at 11:59 PM UTC”.
However, in the submission system homepage, it is
“Paper Upload: June 29, 2018 23:59 CDT”
I want to know, which is the exact one?
thanks,
xin
Hi Xin
We believe the 2nd date should be correct.
Thanks
Zhaohan
Dear Organizers;
As the deadline for submitting the papers is getting close, I would like to ask, that do we have the chance to improve the results that we are going to report in the paper after paper submission? or the model and the parameters must be the same as the one in the paper.
Regards
Hi Sulaiman
Yes, the paper submission right now is for showing an expression of interest and commitment to the challenge. You will definitely have a chance to further improve your method throughout the year.
Thanks
Zhaohan
Dear organizers,
I want to know, for the full paper submission, is it required to be anonymous in the manuscript or not?
thanks,
xin
Hi Xin
The initial manuscript should indeed be anonymous due to the double-blind review process. Detailed instructions can be found here: http://stacom2018.cardiacatlas.org/submission/
Thanks
Zhaohan
Dear organisers,
Is there a page-limit or word-limit for the submission on June 27th?
Regards,
Karim
Hi Rashed
Thanks for your interest. The page limit is 8 pages.
Thanks
Zhaohan
Hi Zhaohan,
Do 8 pages include the reference as well?
BW,
Chen
Hi Chen,
Yes, it should include everything.
Thanks
Zhaohan
Dear Organizer,
thanks for organizing the Challenge and release the data.
As we compare our segmentation results with ground truth, some of the manual labels seem to under-segment the pulmonary vein parts. Our segmentation results sometimes may present longer segmentation than ground truth in segmenting pulmonary veins. Also, we find that the ground truth are not very consistent across the training dataset in labeling the pulmonary veins with different lengths.
Is there any inconsistency in labeling the pulmonary vein? If it is, then how should we address this problem.
thanks,
xin
Hi Xin, thanks for your interest in the challenge.
The data was very carefully manually segmented, with the intent of being consistent across each LGE-MRI. However, due to the sheer scale of the dataset (>100 independent 3D LGE-MRIs), it is very difficult to ensure all the labels are exactly consistent with each other. This is simply a limitation of manual labeling by humans, which we will investigate in future studies. If you could, would you be able to point out exactly which data you found were inconsistent? This will help the organizers gain a clearer idea of the mentioned issues.
Thanks
Zhaohan
Hi,
do we have to submit a preliminary abstract by 17 June 2018, such as required in the STACOM workshop?
Regards,
Sandy
Hi Sandy, thanks for your interest in the challenge.
For the abstract, it is simply a placeholder and can be very simple in content. You are allowed to change it later on in the year, but for now, as a bare minimum, please submit a title, author names and a few sentences briefly describing your methods.
Thanks
Zhaohan
Hi,
I wonder if there is any description for the supporting files. And whether such kind of supporting files will be provided along with the test data in the future.
Hi Chen, thanks for your interest in the challenge.
The supporting files contain labels for whether a patient pre or post ablation, as well as MRA sequences if they are available for the particular patient (as provided in the data section). The pre/post ablation information for the test set is already provided in the support files, as later on, 3D MRIs with the corresponding names in the excel spreadsheet will be uploaded for the test set. With regards to the MRA sequences, we cannot guarantee it as not all of the original 3D MRIs contained MRA sequences. Unfortunately, this was just how the dataset was provided to us. Since the MRA sequences do not correspond to the labels anyways in terms of spatial size, we think that it shouldn’t be of major concern.
Thanks
Zhaohan
Hi,
we noticed that the original spacing of the datasets, which is 0.625 x 0.625 x 1.25 mm³ according to the Data Decription (http://atriaseg2018.cardiacatlas.org/data/), is lost in the provided data sets. They all have isotropic resolution of 1 mm and appear squeezed in z-direction. My guess is that the data was probably not resampled, but the spacing information was just overwritten.
I am wondering how to deal with that: Should we reconstruct the original spacing by scaling/resampling the dataset? Which spacing should we use before applying Run Length Encoding, which is required for submission?
What was your intension to change the spacing? Wouldn’t it be better to provide us data sets with the original spacing in order to avoid confusion?
Regards,
Sandy
Hi Sandy, Thanks for your interest in the challenge.
The meta-information in the original files were over-written to avoid disclosing confidential information. You do not need to do anything in particular in terms of scaling, as run-length encoding does not depend on the original spatial distance of the data, and only depends on the dimensions of the matrix the data occupies (e.g. 640 x 640 x 88). You should perform ur run-length encoding on the MRIs with the given matrix dimensions. The spatial resolution of the MRI is not important in this context, and we expect segmentations to be performed on the dataset format provided to the participants. If you wish, you can obtain the original spacing by sampling every 2nd slice for each 3D scan, however, during submissions, you will still need to provide your segmentation for the 88 slices in the data format.
Thanks
Zhaohan
Hi,
I’m wondering why the paper submission deadline is two months earlier than the competition deadline, should the method detailed in the paper be exactly the same as that submitted to the competition in Sept ? As we know, we can make further improvement after submitting the paper in these two months.
Thanks
Qing Xia
Hi Qing, thanks for your interest in the challenge.
For the paper submission, you can outline the approach you use. Of course, it may be slightly different than the approach in the final competition deadline, but you can update the manuscript/poster/oral presentation to make changes after the deadline. The first draft is simply to give us, the organizers, a general idea of what methods people are using and the preliminary results obtained. You are definitely encouraged to improve your methods after the initial paper submission deadline as we understand many things can change in 2 months.
Thanks
Zhaohan
Thanks for your kindness.
I have one more concern, is the final competition in Sept. only open for those whose paper has been accepted in July?
Thanks
Qing Xia
Hi Qing, thanks again for your question.
The final competition in September is open for all participants, but final awards will be given for those who have submitted their paper and have attended the workshop.
Thanks
Zhaohan
Hi,
I went throughout the literature in order to find some papers as a benchmark for the sake of evaluation of our algorithm, but I couldn’t find something related to atrial fibrillation chamber segmentation. Would you mind to share any paper if you previously published in the same context?
Regards
Hi Sulaiman, thanks for your interest in the challenge.
Atrial segmentation from LGE-MRIs is a very challenging task, and there have not been many successful attempts in previous literature. However, there has been work on atrial segmentation from non-enhanced MRIs (https://ieeexplore.ieee.org/document/7029623/). If you follow the link and read the published work, it should give you a good idea on the methods that were used, however, we strongly advise participants to develop a novel approach since LGE-MRIs are very different than non-enhanced MRIs.
Hope we answered your question.
Thanks
Zhaohan