Discussion Forum

Posts from the Organizers

Recent Comments

43 Comments

  1. Dear Zhaohan Xiong,

    We have completed our prediction, but unfortunately, we do not remember the ID that we registered our team for this challenge. Do you have any alternative solution to resolve this issue?

    • Hi Vesal Sulaiman

      As long as you use the same ID for all your submissions and the ID/email you use is something that is easy for us to recognize who you are, then its fine.

      Thanks

      Zhaohan

  2. Dear Organizers,

    Could you kindly provide some comments on the testing results? We can only get the highest Dice score in the result email, rather than the average Dice over all testing samples. We hope to get the average value, since it should be more informative than the highest one.

    thanks,
    xin

    • Hi Xin

      The value sent in the email is indeed the average DICE over all test samples, not the highest DICE score from any individual test data.

      Thanks

      Zhaohan.

  3. Dear atrial segmentation challenge participants,

    This email is a friendly reminder that our 54 testing data will be released shortly before the starting of the 1 September (UTC) and final competition ends on 14 September (UTC).

    For submission of your final results, check the link below:
    http://atriaseg2018.cardiacatlas.org/submission/

    Important note for final challenge scoring and ranking:
    1. The approaches we seek to evaluate in this challenge need to be automatic (no manual modification);
    2. We will ask for the final five top-ranked teams to submit their codes to us for further validation in addition to their result submission (by 14 Sept). Thus if you think you are doing well for the testing data, better to submit your final results and codes to us at the same time.

    Please note that any researcher/s can participate in the challenge submission/evaluation as well as the bench-marking study following the challenge regardless of whether or not they attend the STACOM conference. However, the monetary prize will only be award based on the best DICE score during the evaluation stage of the challenge and is only available for those who has their paper accepted and presented at the STACOM workshop.

    Any further questions, please post online:
    http://atriaseg2018.cardiacatlas.org/discussion-forum/

  4. Dear Organizers,

    As you have mentioned in submission section: “*It is highly recommended participants use the template submission file as it includes all the rows necessary for prediction.”

    However, the template submission file is the old version and It’s only for 25 patients, not 54.

    • Hi Sulaiman

      The updated template will be provided when the test set is released tomorrow.

      Thanks.
      Zhaohan.

  5. Dear Organizers,

    we just received emails from STACOM about the release of all submitted papers. Why the papers full of technical details and results are released before the final competition?
    At the beginning of the Challenge, the rule is that “no paper submission, no final testing”. Then, the rule is changed as in the “Paper Review Comments” section, “For those who did not submit a paper, we still encourage you to submit entries for the challenge, as you will be scored on the same leaderboard.”
    We can accept this kind of rule change. However, releasing the paper of teams before the final testing is definitely unfair. Those teams who didn’t submit their papers can potentially benefit from those released papers with respect to methods and results, while they don’t need to disclose their methods and results for other teams for reference.
    This is definitely unfair for the teams who have prepared for this Challenge for months.
    We hope the organizers can explain on this problem.

    thanks,
    xin

    • Hi Xin

      I have already replied to you via email but I think this will benefit other participants. We do not have control of conference paper publishing so we will address this issue with the conference organizers. Also, the people who did not submit a paper will only be scored, and will not receive the prize regardless of their score. This rule remained the same throughout the challenge and we did not change it.

      Thanks.
      Zhaohan.

    • many thanks for your email and concern, apologize if it is not clear enough since it is really our job to make it fair, open and encourage more people to participate.

      THE RULE for our challenge is same and consistent ALWAYs: that is to claim prize and place in the final challenge, one will need to submit the MICCAI paper and present there in STACOM either in poster or oral. However, one misses the paper submission, can still submit their results to us for evaluation, and may have a chance to participate the bench-marking study as an joint journal paper (as well as high performances of the MICCAI participants). However, we do agree we may change the wording a little bit in different stages, but the RULE is same.

      When to release the drafted paper is also same and consistent year after year by the MICCAI conference, I am not sure we can stop them for sharing prior to conference, but I will send an email for asking them anyway. Secondly, everyone participating the atrial challenge is sharing their papers at the same time, we will not see any problem here. Since everyone is using different approaches, if someone does copy your approach, we are sure we can use the originally submitted paper to trace it.

      On the contrary, to make the challenge more fair, we are doing everything we can to improve it
      1) by increasing more testing data to 54 3D LGE-MRI; and
      2) asking the final 5 top ranked teams to submit their codes for further validating in addition to their result submission.

      Finally, we would advise everyone to focus on improving their approaches instead of worrying something absolutely unnecessary in your final days.

      many thanks,
      sincerely
      Jichao ZHAO

  6. Dear organizers,

    can you tell us how many submissions you have received for the challenge? It is interesting to know how many teams actually participate.

    Regards,
    Sandy

    • Hi Sandy

      There have been over 150 different individuals who have accessed our data. We have had 17 conference paper submissions. With this being said, we are still not sure how many exactly will participate in the final evaluation stage of the challenge. Hope this information helps nevertheless.

      Thanks
      Zhaohan

  7. Dear Organizers,

    as the review process has finished, we want to know, when can we get the detailed comments from reviewers? since it’s only about 1 week before the camera-ready deadline.

    thanks,
    xin

    • Hi Xin

      We are working towards getting feedback, however, we will still need to discuss with the STACOM organizers to give a final review. Please be patient as there were a lot of submissions, particularly for this challenge.

      Thanks
      Zhaohan

  8. Hi,
    Because the test set is open too late, I would like to know if the organizer can publish the best result of the validationset according to the paper results.Although the reliability of the result is not too high, because of the difference of the validation, it still has certain reference value for us.
    Thanks,
    Jianqiang Ma

    • Hi Jianqiang

      The reason we are only allowing 2 weeks for participants to predict for the test set is to avoid manual labeling as if we release it months prior, then it would be easier to manually segment the atria, which is not the point of the competition. Unfortunately, it is not possible to publish all the results from the preliminary validation, as the difference in validation strategies results in vastly different scores which might be confusing for some people. However, I can tell you that some teams have achieved roughly 0.90 DICE scores when performing validation on the training sets.

      Thanks

      Zhaohan

  9. Hi,
    I’m wondering why the paper submission website just has submission abstract button. Is the paper submitted when I submit the paper in submission abstract? I hope you can understand me.
    Thanks.

    • Hi Yashu

      According to the organizers of STACOM, you can:
      1) register the abstract and upload the paper a (browse document PDF) at that time (see attached); or
      2) register and later click on the title of the paper, and you should have a green button at the bottom to click for upload of the paper (see snapshot). You can also re-upload a new PDF version which will override the previous one.

      Thanks
      Zhaohan

  10. Dear Organizers,

    apologize to bother you again.
    for the full paper submission deadline, there seems to be some inconsistency.
    On the Challenge website, the full paper deadline is
    “28 June 2018 at 11:59 PM UTC”.
    However, in the submission system homepage, it is
    “Paper Upload: June 29, 2018 23:59 CDT”
    I want to know, which is the exact one?

    thanks,
    xin

  11. Dear Organizers;

    As the deadline for submitting the papers is getting close, I would like to ask, that do we have the chance to improve the results that we are going to report in the paper after paper submission? or the model and the parameters must be the same as the one in the paper.

    Regards

    • Hi Sulaiman

      Yes, the paper submission right now is for showing an expression of interest and commitment to the challenge. You will definitely have a chance to further improve your method throughout the year.

      Thanks
      Zhaohan

  12. Dear organizers,

    I want to know, for the full paper submission, is it required to be anonymous in the manuscript or not?

    thanks,
    xin

  13. Dear organisers,

    Is there a page-limit or word-limit for the submission on June 27th?

    Regards,
    Karim

  14. Dear Organizer,

    thanks for organizing the Challenge and release the data.
    As we compare our segmentation results with ground truth, some of the manual labels seem to under-segment the pulmonary vein parts. Our segmentation results sometimes may present longer segmentation than ground truth in segmenting pulmonary veins. Also, we find that the ground truth are not very consistent across the training dataset in labeling the pulmonary veins with different lengths.
    Is there any inconsistency in labeling the pulmonary vein? If it is, then how should we address this problem.

    thanks,
    xin

    • Hi Xin, thanks for your interest in the challenge.

      The data was very carefully manually segmented, with the intent of being consistent across each LGE-MRI. However, due to the sheer scale of the dataset (>100 independent 3D LGE-MRIs), it is very difficult to ensure all the labels are exactly consistent with each other. This is simply a limitation of manual labeling by humans, which we will investigate in future studies. If you could, would you be able to point out exactly which data you found were inconsistent? This will help the organizers gain a clearer idea of the mentioned issues.

      Thanks
      Zhaohan

  15. Hi,

    do we have to submit a preliminary abstract by 17 June 2018, such as required in the STACOM workshop?

    Regards,
    Sandy

    • Hi Sandy, thanks for your interest in the challenge.

      For the abstract, it is simply a placeholder and can be very simple in content. You are allowed to change it later on in the year, but for now, as a bare minimum, please submit a title, author names and a few sentences briefly describing your methods.

      Thanks
      Zhaohan

  16. Hi,

    I wonder if there is any description for the supporting files. And whether such kind of supporting files will be provided along with the test data in the future.

    • Hi Chen, thanks for your interest in the challenge.

      The supporting files contain labels for whether a patient pre or post ablation, as well as MRA sequences if they are available for the particular patient (as provided in the data section). The pre/post ablation information for the test set is already provided in the support files, as later on, 3D MRIs with the corresponding names in the excel spreadsheet will be uploaded for the test set. With regards to the MRA sequences, we cannot guarantee it as not all of the original 3D MRIs contained MRA sequences. Unfortunately, this was just how the dataset was provided to us. Since the MRA sequences do not correspond to the labels anyways in terms of spatial size, we think that it shouldn’t be of major concern.

      Thanks
      Zhaohan

  17. Hi,

    we noticed that the original spacing of the datasets, which is 0.625 x 0.625 x 1.25 mm³ according to the Data Decription (http://atriaseg2018.cardiacatlas.org/data/), is lost in the provided data sets. They all have isotropic resolution of 1 mm and appear squeezed in z-direction. My guess is that the data was probably not resampled, but the spacing information was just overwritten.
    I am wondering how to deal with that: Should we reconstruct the original spacing by scaling/resampling the dataset? Which spacing should we use before applying Run Length Encoding, which is required for submission?

    What was your intension to change the spacing? Wouldn’t it be better to provide us data sets with the original spacing in order to avoid confusion?

    Regards,
    Sandy

    • Hi Sandy, Thanks for your interest in the challenge.

      The meta-information in the original files were over-written to avoid disclosing confidential information. You do not need to do anything in particular in terms of scaling, as run-length encoding does not depend on the original spatial distance of the data, and only depends on the dimensions of the matrix the data occupies (e.g. 640 x 640 x 88). You should perform ur run-length encoding on the MRIs with the given matrix dimensions. The spatial resolution of the MRI is not important in this context, and we expect segmentations to be performed on the dataset format provided to the participants. If you wish, you can obtain the original spacing by sampling every 2nd slice for each 3D scan, however, during submissions, you will still need to provide your segmentation for the 88 slices in the data format.

      Thanks
      Zhaohan

  18. Hi,

    I’m wondering why the paper submission deadline is two months earlier than the competition deadline, should the method detailed in the paper be exactly the same as that submitted to the competition in Sept ? As we know, we can make further improvement after submitting the paper in these two months.

    Thanks
    Qing Xia

    • Hi Qing, thanks for your interest in the challenge.

      For the paper submission, you can outline the approach you use. Of course, it may be slightly different than the approach in the final competition deadline, but you can update the manuscript/poster/oral presentation to make changes after the deadline. The first draft is simply to give us, the organizers, a general idea of what methods people are using and the preliminary results obtained. You are definitely encouraged to improve your methods after the initial paper submission deadline as we understand many things can change in 2 months.

      Thanks
      Zhaohan

      • Thanks for your kindness.

        I have one more concern, is the final competition in Sept. only open for those whose paper has been accepted in July?

        Thanks
        Qing Xia

        • Hi Qing, thanks again for your question.

          The final competition in September is open for all participants, but final awards will be given for those who have submitted their paper and have attended the workshop.

          Thanks
          Zhaohan

  19. Hi,

    I went throughout the literature in order to find some papers as a benchmark for the sake of evaluation of our algorithm, but I couldn’t find something related to atrial fibrillation chamber segmentation. Would you mind to share any paper if you previously published in the same context?

    Regards

    • Hi Sulaiman, thanks for your interest in the challenge.

      Atrial segmentation from LGE-MRIs is a very challenging task, and there have not been many successful attempts in previous literature. However, there has been work on atrial segmentation from non-enhanced MRIs (https://ieeexplore.ieee.org/document/7029623/). If you follow the link and read the published work, it should give you a good idea on the methods that were used, however, we strongly advise participants to develop a novel approach since LGE-MRIs are very different than non-enhanced MRIs.

      Hope we answered your question.

      Thanks

      Zhaohan

Comments are closed.