You don't needed to select any category in JComments settings. You have to enable commenting in JDownloads and select JComments as commenting extension in JDownloads settings.
If you're want to comment categories (not files) you have to make small modification of JDownloads:
1. Open /components/com_jdownloads/jdownloads.html.php
2. Find code:
$footer = makeFooter(true, false, true, $sum_pages, $limit, $limitstart, $site_aktuell, $pageNav, false, false);
$html_cat .= $footer;
// remove empty html tags
if ($jlistConfig['remove.empty.tags']){
$html_cat = removeEmptyTags($html_cat);
}
and replace with:
$footer = makeFooter(true, false, true, $sum_pages, $limit, $limitstart, $site_aktuell, $pageNav, false, false);
if ($jlistConfig['jcomments.active']){
$jcomments = $mainframe->getCfg('absolute_path') . '/components/com_jcomments/jcomments.php';
if (file_exists($jcomments)) {
require_once($jcomments);
$obj_id = $cat[0]->cat_id;
$obj_title = $cat[0]->cat_title;
$html_cat .= JComments::showComments($obj_id, 'com_jdownloads_category', $obj_title);
}
}
$html_cat .= $footer;
// remove empty html tags
if ($jlistConfig['remove.empty.tags']){
$html_cat = removeEmptyTags($html_cat);
}
I haven't tested this modification but seems it should work

So try to apply this modification and if comments form will show under category - I will explain what to do after this.