Install the following packages in the environment
conda install vispy astropy traitlets bqplot bqplot-image-gl glue-core ipyvue ipyvuetify voila pyyaml specutils glue-astronomy click asteval idna asdf gwcs regions scikit-image sidecar conda install et-xmlfile freetype-py hsluv ipydatawidgets ipympl ipyvolume ipywebrtc openpyxl pyopengl pythreejs
Then you can just install it with pip
pip install jdaviz --upgrade
Did not work out so in the end I installed everything through pip.
conda create -n py3-jdaviz python=3.8 conda activate py3-jdaviz pip install vispy>=0.6.5 pip install jdaviz --no-cache-dir