Hello

(First message on groups.io here: I hope I'm following locale rules, if not, don't hesitate to correct me !)

Since october 2022, Fiona's wheels include GDAL (according to the releases documentations). Many packages refer to GDAL using this command, but it won't work:

from osgeo import gdal

For instance, I've just loaded an environment using poetry (and python 3.9.15 on Linux):

poetry new dummy
cd dummy
poetry add geopandas richdem

Calling this script will throw an error:

import richdem as rd
dem = rd.LoadGDAL("dummy")
> Exception: richdem.LoadGDAL() requires GDAL.

I also tried to load Fiona at first (it patches GDAL environment variables at launch, if I've understood it correctly), but it doesn't change anything.

Note:
If I now install gdal (same version as the one included in my 1.8.22 Fiona, ie. gdal 3.4.3), then I ran into a different exception; I suspect it may be linked to poetry and the steps to configure my virtual environment:

rd.LoadGDAL("a/real/tif_file.tif")
> ModuleNotFoundError: No module named '_gdal_array'

So the question is:
how could I fix the `from osgeo import gdal` from third party packages calling for GDAL (if GDAL is embedded in Fiona)?

The Fiona wheels do not include the GDAL Python bindings (which you try to import with "from osgeo import gdal").

If you need the GDAL Python bindings you will find information on how to install them at https://gdal.org/api/python_bindings.html 

 

Thanks.

So when Fiona's release states that 1.9a3 wheels "include GDAL 3.5.2, PROJ 9.0.1, and GEOS 3.11.0", what do they really include? The C++ core of GDAL?

I mean if I have to reinstall the GDAL package (which I allready tried, but ended in another error, possibly linked to a misconfiguration of virtual environement), won't I end up with two GDAL cores? How can that be a good solution?

(Note that I'm not critical of Fiona including GDAL; I'm 100% gratefull for it as it was my major trouble for switching between linux and windows)

On Tue, 29 Nov 2022 at 11:27, <tgrandje@...> wrote:

Thanks.

So when Fiona's release states that 1.9a3 wheels "include GDAL 3.5.2, PROJ 9.0.1, and GEOS 3.11.0", what do they really include? The C++ core of GDAL?

I mean if I have to reinstall the GDAL package (which I allready tried, but ended in another error, possibly linked to a misconfiguration of virtual environement), won't I end up with two GDAL cores? How can that be a good solution?

If you want to have both Fiona and the GDAL python bindings installed,
you cannot do that using pip with binary wheels. Possible solution are
to installing those packages from source (building them yourself), or
to use the conda package manager.

Joris

(Note that I'm not critical of Fiona including GDAL; I'm 100% gratefull for it as it was my major trouble for switching between linux and windows)

I'm gratefull for the comments, though I'm still lost.

It does not help that Fiona's homepage (on pypi) seems out of date (for instance, last test I did on windows, everything went fine with only a "pip install Fiona", including using other packages like rasterstats). Should I raise an issue on Github or do I have it all wrong (I'm still not clear on what is embedded in Fiona wheels) ?

In any case, I'd say that those remarks about building from source when requiring both osgeo and fiona needs to be stated in documentation. I mean by that that people using Fiona are prone to use other packages built on GDAL. (I suspect it also means I can't build a package with poetry if both GDAL and Fiona are required.)

And to sum it up regarding to my first question: so there is no way to change those packages to substitute a Fiona dependency rather than an osgeo call ?

On Tue, 29 Nov 2022 at 11:55, <tgrandje@...> wrote:

I'm gratefull for the comments, though I'm still lost.

It does not help that Fiona's homepage (on pypi) seems out of date (for instance, last test I did on windows, everything went fine with only a "pip install Fiona", including using other packages like rasterstats). Should I raise an issue on Github or do I have it all wrong (I'm still not clear on what is embedded in Fiona wheels) ?

That indeed seems outdated (there are now wheels for Windows as well),
so it would be good to raise an issue about that.

In any case, I'd say that those remarks about building from source when requiring both osgeo and fiona needs to be stated in documentation. I mean by that that people using Fiona are prone to use other packages built on GDAL. (I suspect it also means I can't build a package with poetry if both GDAL and Fiona are required.)

And to sum it up regarding to my first question: so there is no way to change those packages to substitute a Fiona dependency rather than an osgeo call ?

No, the APIs of Fiona or gdal python bindings are completely
different, so it's up to the package to make a choice which one to
take as a dependency.

Fiona's wheels contain a GDAL shared library (libgdal.dll or .so or .dylib) and its own library dependencies (libproj, etc). By GDAL, we mean the C library. I don't think this is inaccurate. I don't see that it's completely natural to assume that fiona imports osgeo.gdal. There are no such imports anywhere in the code and no instructions in the project to install the osgeo.gdal package.

@Sean Gillies: sure, I made the wrong assumption. Nonetheless, you can't help to jump to conclusions when you see that info in the release on pypi, that you're 100% python coder and that GDAL is also the name of the python binding's package... Everything about GDAL shared library is running under the hood, so I don't even have to understand (very much) how it works. I'd say it would'nt be out of place to state all of this on pypi's homepage. I'll raise an issue about that on GitHub as well as elements disscussed with @Joris Van den Bossche.

It's not a great solution, I agree, and not only because you'll have two copies of libgdal. The Fiona project is not promoting it as a solution at all. In https://rasterio.readthedocs.io/en/latest/topics/switch.html#mutual-incompatibilities, I've warned users against this and I should probably do the same for Fiona to make it more clear. Would that help?

@Sean Gillies: Yes, definitively. You see, thanks to the amazing work on geopandas, I don't even use Fiona - except under the hood to manipulate files with geopandas. It's not easy to track all those limitations when it involves sub-dependencies. For instance, I knew of the warning on rasterio (though the warning may have been completed since I last read it and I certainly didn't make the connection with my present case); what I haven't figured yet is if it's safe to use geopandas and rasterio at the same time (which would the obvious thing to do if you wan't to use something like rasterstats, if there wasn't this incompatibility warning). So far, I haven't found inconsistencies on mixing those, but maybe there is. If that is, maybe a warning should also be set on geopandas (if not, I'd also said it would be adequate just to be explicite). If you have the answer to that question, I'd be gratefull to have it (in that case, I understand it could be safe to have the double installation of gdal, if I ever manage to configure a running virtualenv with both...)

Well in any case, thanks for those valuable insights!