#10 Allow overriding sys.platlibdir via env-var
Closed 3 years ago by smani. Opened 3 years ago by smani.
Unknown source master  into  master

@@ -0,0 +1,346 @@

+ diff -rupN Python-3.9.0b1/Doc/c-api/init_config.rst Python-3.9.0b1-new/Doc/c-api/init_config.rst

+ --- Python-3.9.0b1/Doc/c-api/init_config.rst	2020-05-19 01:07:09.000000000 +0200

+ +++ Python-3.9.0b1-new/Doc/c-api/init_config.rst	2020-06-08 12:13:23.639148262 +0200

+ @@ -436,6 +436,14 @@ PyConfig

+  

+        :data:`sys.base_prefix`.

+  

+ +   .. c:member:: wchar_t* platlibdir

+ +

+ +      :data:`sys.platlibdir`: platform library directory name, set at configure time

+ +      by ``--with-platlibdir``, overrideable by the ``PYTHONPLATLIBDIR``

+ +      environment variable.

+ +

+ +      .. versionadded:: 3.10

+ +

+     .. c:member:: int buffered_stdio

+  

+        If equals to 0, enable unbuffered mode, making the stdout and stderr

+ @@ -884,6 +892,7 @@ Path Configuration

+  * Path configuration inputs:

+  

+    * :c:member:`PyConfig.home`

+ +  * :c:member:`PyConfig.platlibdir`

+    * :c:member:`PyConfig.pathconfig_warnings`

+    * :c:member:`PyConfig.program_name`

+    * :c:member:`PyConfig.pythonpath_env`

+ diff -rupN Python-3.9.0b1/Doc/using/cmdline.rst Python-3.9.0b1-new/Doc/using/cmdline.rst

+ --- Python-3.9.0b1/Doc/using/cmdline.rst	2020-05-19 01:07:09.000000000 +0200

+ +++ Python-3.9.0b1-new/Doc/using/cmdline.rst	2020-06-08 12:13:23.639148262 +0200

+ @@ -538,6 +538,14 @@ conflict.

+     within a Python program as the variable :data:`sys.path`.

+  

+  

+ +.. envvar:: PYTHONPLATLIBDIR

+ +

+ +   If this is set to a non-empty string, it overrides the :data:`sys.platlibdir`

+ +   value.

+ +

+ +   .. versionadded:: 3.10

+ +

+ +

+  .. envvar:: PYTHONSTARTUP

+  

+     If this is the name of a readable file, the Python commands in that file are

+ diff -rupN Python-3.9.0b1/Include/cpython/initconfig.h Python-3.9.0b1-new/Include/cpython/initconfig.h

+ --- Python-3.9.0b1/Include/cpython/initconfig.h	2020-05-19 01:07:09.000000000 +0200

+ +++ Python-3.9.0b1-new/Include/cpython/initconfig.h	2020-06-08 12:13:23.640148261 +0200

+ @@ -388,6 +388,7 @@ typedef struct {

+      wchar_t *base_prefix;       /* sys.base_prefix */

+      wchar_t *exec_prefix;       /* sys.exec_prefix */

+      wchar_t *base_exec_prefix;  /* sys.base_exec_prefix */

+ +    wchar_t *platlibdir;        /* sys.platlibdir */

+  

+      /* --- Parameter only used by Py_Main() ---------- */

+  

+ diff -rupN Python-3.9.0b1/Lib/test/test_embed.py Python-3.9.0b1-new/Lib/test/test_embed.py

+ --- Python-3.9.0b1/Lib/test/test_embed.py	2020-05-19 01:07:09.000000000 +0200

+ +++ Python-3.9.0b1-new/Lib/test/test_embed.py	2020-06-08 12:13:23.641148261 +0200

+ @@ -380,6 +380,7 @@ class InitConfigTests(EmbeddingTestsMixi

+          'exec_prefix': GET_DEFAULT_CONFIG,

+          'base_exec_prefix': GET_DEFAULT_CONFIG,

+          'module_search_paths': GET_DEFAULT_CONFIG,

+ +        'platlibdir': sys.platlibdir,

+  

+          'site_import': 1,

+          'bytes_warning': 0,

+ @@ -585,13 +586,14 @@ class InitConfigTests(EmbeddingTestsMixi

+              if value is self.GET_DEFAULT_CONFIG:

+                  expected[key] = config[key]

+  

+ -        pythonpath_env = expected['pythonpath_env']

+ -        if pythonpath_env is not None:

+ -            paths = pythonpath_env.split(os.path.pathsep)

+ -            expected['module_search_paths'] = [*paths, *expected['module_search_paths']]

+ -        if modify_path_cb is not None:

+ -            expected['module_search_paths'] = expected['module_search_paths'].copy()

+ -            modify_path_cb(expected['module_search_paths'])

+ +        if expected['module_search_paths'] is not self.IGNORE_CONFIG:

+ +            pythonpath_env = expected['pythonpath_env']

+ +            if pythonpath_env is not None:

+ +                paths = pythonpath_env.split(os.path.pathsep)

+ +                expected['module_search_paths'] = [*paths, *expected['module_search_paths']]

+ +            if modify_path_cb is not None:

+ +                expected['module_search_paths'] = expected['module_search_paths'].copy()

+ +                modify_path_cb(expected['module_search_paths'])

+  

+          for key in self.COPY_PRE_CONFIG:

+              if key not in expected_preconfig:

+ @@ -764,6 +766,8 @@ class InitConfigTests(EmbeddingTestsMixi

+              'buffered_stdio': 0,

+              'user_site_directory': 0,

+              'faulthandler': 1,

+ +            'platlibdir': 'my_platlibdir',

+ +            'module_search_paths': self.IGNORE_CONFIG,

+  

+              'check_hash_pycs_mode': 'always',

+              'pathconfig_warnings': 0,

+ @@ -795,6 +799,8 @@ class InitConfigTests(EmbeddingTestsMixi

+              'user_site_directory': 0,

+              'faulthandler': 1,

+              'warnoptions': ['EnvVar'],

+ +            'platlibdir': 'env_platlibdir',

+ +            'module_search_paths': self.IGNORE_CONFIG,

+              '_use_peg_parser': 0,

+          }

+          self.check_all_configs("test_init_compat_env", config, preconfig,

+ @@ -823,6 +829,8 @@ class InitConfigTests(EmbeddingTestsMixi

+              'user_site_directory': 0,

+              'faulthandler': 1,

+              'warnoptions': ['EnvVar'],

+ +            'platlibdir': 'env_platlibdir',

+ +            'module_search_paths': self.IGNORE_CONFIG,

+              '_use_peg_parser': 0,

+          }

+          self.check_all_configs("test_init_python_env", config, preconfig,

+ diff -rupN Python-3.9.0b1/Makefile.pre.in Python-3.9.0b1-new/Makefile.pre.in

+ --- Python-3.9.0b1/Makefile.pre.in	2020-05-19 01:07:09.000000000 +0200

+ +++ Python-3.9.0b1-new/Makefile.pre.in	2020-06-08 12:13:23.641148261 +0200

+ @@ -811,6 +811,11 @@ Python/sysmodule.o: $(srcdir)/Python/sys

+  		$(MULTIARCH_CPPFLAGS) \

+  		-o $@ $(srcdir)/Python/sysmodule.c

+  

+ +Python/initconfig.o: $(srcdir)/Python/initconfig.c

+ +	$(CC) -c $(PY_CORE_CFLAGS) \

+ +		-DPLATLIBDIR='"$(PLATLIBDIR)"' \

+ +		-o $@ $(srcdir)/Python/initconfig.c

+ +

+  $(IO_OBJS): $(IO_H)

+  

+  .PHONY: regen-grammar

+ diff -rupN "Python-3.9.0b1/Misc/NEWS.d/next/Core and Builtins/2020-06-03-13-53-24.bpo-40854.O6vfQU.rst" "Python-3.9.0b1-new/Misc/NEWS.d/next/Core and Builtins/2020-06-03-13-53-24.bpo-40854.O6vfQU.rst"

+ --- "Python-3.9.0b1/Misc/NEWS.d/next/Core and Builtins/2020-06-03-13-53-24.bpo-40854.O6vfQU.rst"	1970-01-01 01:00:00.000000000 +0100

+ +++ "Python-3.9.0b1-new/Misc/NEWS.d/next/Core and Builtins/2020-06-03-13-53-24.bpo-40854.O6vfQU.rst"	2020-06-08 12:13:23.642148260 +0200

+ @@ -0,0 +1 @@

+ +Allow overriding :data:`sys.platlibdir` via a new :envvar:`PYTHONPLATLIBDIR` environment variable.

+ diff -rupN Python-3.9.0b1/Misc/python.man Python-3.9.0b1-new/Misc/python.man

+ --- Python-3.9.0b1/Misc/python.man	2020-05-19 01:07:09.000000000 +0200

+ +++ Python-3.9.0b1-new/Misc/python.man	2020-06-08 12:13:23.643148260 +0200

+ @@ -413,6 +413,8 @@ inserted in the path in front of $PYTHON

+  The search path can be manipulated from within a Python program as the

+  variable

+  .IR sys.path .

+ +.IP PYTHONPLATLIBDIR

+ +Override sys.platlibdir.

+  .IP PYTHONSTARTUP

+  If this is the name of a readable file, the Python commands in that

+  file are executed before the first prompt is displayed in interactive

+ diff -rupN Python-3.9.0b1/Modules/getpath.c Python-3.9.0b1-new/Modules/getpath.c

+ --- Python-3.9.0b1/Modules/getpath.c	2020-05-19 01:07:09.000000000 +0200

+ +++ Python-3.9.0b1-new/Modules/getpath.c	2020-06-08 12:13:55.800135297 +0200

+ @@ -105,8 +105,8 @@ extern "C" {

+  

+  

+  #if (!defined(PREFIX) || !defined(EXEC_PREFIX) \

+ -        || !defined(VERSION) || !defined(VPATH) || !defined(PLATLIBDIR))

+ -#error "PREFIX, EXEC_PREFIX, VERSION, VPATH and PLATLIBDIR macros must be defined"

+ +        || !defined(VERSION) || !defined(VPATH))

+ +#error "PREFIX, EXEC_PREFIX, VERSION and VPATH macros must be defined"

+  #endif

+  

+  #ifndef LANDMARK

+ @@ -128,7 +128,6 @@ typedef struct {

+      wchar_t *pythonpath_macro;         /* PYTHONPATH macro */

+      wchar_t *prefix_macro;             /* PREFIX macro */

+      wchar_t *exec_prefix_macro;        /* EXEC_PREFIX macro */

+ -    wchar_t *platlibdir_macro;         /* PLATLIBDIR macro */

+      wchar_t *vpath_macro;              /* VPATH macro */

+  

+      wchar_t *lib_python;               /* "lib/pythonX.Y" */

+ @@ -138,6 +137,7 @@ typedef struct {

+  

+      int warnings;

+      const wchar_t *pythonpath_env;

+ +    const wchar_t *platlibdir;

+  

+      wchar_t *argv0_path;

+      wchar_t *zip_path;

+ @@ -811,7 +811,7 @@ calculate_exec_prefix(PyCalculatePath *c

+          }

+  

+          /* <PLATLIBDIR> / "lib-dynload" */

+ -        wchar_t *lib_dynload = joinpath2(calculate->platlibdir_macro,

+ +        wchar_t *lib_dynload = joinpath2(calculate->platlibdir,

+                                           L"lib-dynload");

+          if (lib_dynload == NULL) {

+              return _PyStatus_NO_MEMORY();

+ @@ -1297,7 +1297,7 @@ calculate_zip_path(PyCalculatePath *calc

+      PyStatus res;

+  

+      /* Path: <PLATLIBDIR> / "python00.zip" */

+ -    wchar_t *path = joinpath2(calculate->platlibdir_macro, L"python00.zip");

+ +    wchar_t *path = joinpath2(calculate->platlibdir, L"python00.zip");

+      if (path == NULL) {

+          return _PyStatus_NO_MEMORY();

+      }

+ @@ -1456,10 +1456,6 @@ calculate_init(PyCalculatePath *calculat

+      if (!calculate->vpath_macro) {

+          return DECODE_LOCALE_ERR("VPATH macro", len);

+      }

+ -    calculate->platlibdir_macro = Py_DecodeLocale(PLATLIBDIR, &len);

+ -    if (!calculate->platlibdir_macro) {

+ -        return DECODE_LOCALE_ERR("PLATLIBDIR macro", len);

+ -    }

+  

+      calculate->lib_python = Py_DecodeLocale(PLATLIBDIR "/python" VERSION, &len);

+      if (!calculate->lib_python) {

+ @@ -1468,6 +1464,7 @@ calculate_init(PyCalculatePath *calculat

+  

+      calculate->warnings = config->pathconfig_warnings;

+      calculate->pythonpath_env = config->pythonpath_env;

+ +    calculate->platlibdir = config->platlibdir;

+  

+      return _PyStatus_OK();

+  }

+ @@ -1480,7 +1477,6 @@ calculate_free(PyCalculatePath *calculat

+      PyMem_RawFree(calculate->prefix_macro);

+      PyMem_RawFree(calculate->exec_prefix_macro);

+      PyMem_RawFree(calculate->vpath_macro);

+ -    PyMem_RawFree(calculate->platlibdir_macro);

+      PyMem_RawFree(calculate->lib_python);

+      PyMem_RawFree(calculate->path_env);

+      PyMem_RawFree(calculate->zip_path);

+ diff -rupN Python-3.9.0b1/Programs/_testembed.c Python-3.9.0b1-new/Programs/_testembed.c

+ --- Python-3.9.0b1/Programs/_testembed.c	2020-05-19 01:07:09.000000000 +0200

+ +++ Python-3.9.0b1-new/Programs/_testembed.c	2020-06-08 12:13:23.645148259 +0200

+ @@ -548,6 +548,13 @@ static int test_init_from_config(void)

+      /* FIXME: test home */

+      /* FIXME: test path config: module_search_path .. dll_path */

+  

+ +    putenv("PYTHONPLATLIBDIR=env_platlibdir");

+ +    status = PyConfig_SetBytesString(&config, &config.platlibdir, "my_platlibdir");

+ +    if (PyStatus_Exception(status)) {

+ +        PyConfig_Clear(&config);

+ +        Py_ExitStatusException(status);

+ +    }

+ +

+      putenv("PYTHONVERBOSE=0");

+      Py_VerboseFlag = 0;

+      config.verbose = 1;

+ @@ -668,6 +675,7 @@ static void set_most_env_vars(void)

+      putenv("PYTHONFAULTHANDLER=1");

+      putenv("PYTHONIOENCODING=iso8859-1:replace");

+      putenv("PYTHONOLDPARSER=1");

+ +    putenv("PYTHONPLATLIBDIR=env_platlibdir");

+  }

+  

+  

+ diff -rupN Python-3.9.0b1/Python/initconfig.c Python-3.9.0b1-new/Python/initconfig.c

+ --- Python-3.9.0b1/Python/initconfig.c	2020-05-19 01:07:09.000000000 +0200

+ +++ Python-3.9.0b1-new/Python/initconfig.c	2020-06-08 12:13:23.646148258 +0200

+ @@ -110,6 +110,7 @@ PYTHONPATH   : '%lc'-separated list of d

+  static const char usage_5[] =

+  "PYTHONHOME   : alternate <prefix> directory (or <prefix>%lc<exec_prefix>).\n"

+  "               The default module search path uses %s.\n"

+ +"PYTHONPLATLIBDIR : override sys.platlibdir.\n"

+  "PYTHONCASEOK : ignore case in 'import' statements (Windows).\n"

+  "PYTHONUTF8: if set to 1, enable the UTF-8 mode.\n"

+  "PYTHONIOENCODING: Encoding[:errors] used for stdin/stdout/stderr.\n"

+ @@ -588,6 +589,7 @@ PyConfig_Clear(PyConfig *config)

+      CLEAR(config->base_prefix);

+      CLEAR(config->exec_prefix);

+      CLEAR(config->base_exec_prefix);

+ +    CLEAR(config->platlibdir);

+  

+      CLEAR(config->filesystem_encoding);

+      CLEAR(config->filesystem_errors);

+ @@ -824,6 +826,7 @@ _PyConfig_Copy(PyConfig *config, const P

+      COPY_WSTR_ATTR(base_prefix);

+      COPY_WSTR_ATTR(exec_prefix);

+      COPY_WSTR_ATTR(base_exec_prefix);

+ +    COPY_WSTR_ATTR(platlibdir);

+  

+      COPY_ATTR(site_import);

+      COPY_ATTR(bytes_warning);

+ @@ -926,6 +929,7 @@ config_as_dict(const PyConfig *config)

+      SET_ITEM_WSTR(base_prefix);

+      SET_ITEM_WSTR(exec_prefix);

+      SET_ITEM_WSTR(base_exec_prefix);

+ +    SET_ITEM_WSTR(platlibdir);

+      SET_ITEM_INT(site_import);

+      SET_ITEM_INT(bytes_warning);

+      SET_ITEM_INT(inspect);

+ @@ -1336,6 +1340,14 @@ config_read_env_vars(PyConfig *config)

+          }

+      }

+  

+ +    if(config->platlibdir == NULL) {

+ +        status = CONFIG_GET_ENV_DUP(config, &config->platlibdir,

+ +                                    L"PYTHONPLATLIBDIR", "PYTHONPLATLIBDIR");

+ +        if (_PyStatus_EXCEPTION(status)) {

+ +            return status;

+ +        }

+ +    }

+ +

+      if (config->use_hash_seed < 0) {

+          status = config_init_hash_seed(config);

+          if (_PyStatus_EXCEPTION(status)) {

+ @@ -1731,6 +1743,14 @@ config_read(PyConfig *config)

+          }

+      }

+  

+ +    if(config->platlibdir == NULL) {

+ +        status = CONFIG_SET_BYTES_STR(config, &config->platlibdir, PLATLIBDIR,

+ +                                      "PLATLIBDIR macro");

+ +        if (_PyStatus_EXCEPTION(status)) {

+ +            return status;

+ +        }

+ +    }

+ +

+      if (config->_install_importlib) {

+          status = _PyConfig_InitPathConfig(config);

+          if (_PyStatus_EXCEPTION(status)) {

+ @@ -2554,6 +2574,7 @@ PyConfig_Read(PyConfig *config)

+          assert(config->exec_prefix != NULL);

+          assert(config->base_exec_prefix != NULL);

+      }

+ +    assert(config->platlibdir != NULL);

+      assert(config->filesystem_encoding != NULL);

+      assert(config->filesystem_errors != NULL);

+      assert(config->stdio_encoding != NULL);

+ @@ -2704,6 +2725,7 @@ _Py_DumpPathConfig(PyThreadState *tstate

+      DUMP_SYS(_base_executable);

+      DUMP_SYS(base_prefix);

+      DUMP_SYS(base_exec_prefix);

+ +    DUMP_SYS(platlibdir);

+      DUMP_SYS(executable);

+      DUMP_SYS(prefix);

+      DUMP_SYS(exec_prefix);

+ diff -rupN Python-3.9.0b1/Python/sysmodule.c Python-3.9.0b1-new/Python/sysmodule.c

+ --- Python-3.9.0b1/Python/sysmodule.c	2020-05-19 01:07:09.000000000 +0200

+ +++ Python-3.9.0b1-new/Python/sysmodule.c	2020-06-08 12:13:23.647148258 +0200

+ @@ -2922,13 +2922,7 @@ _PySys_InitMain(PyThreadState *tstate)

+      SET_SYS_FROM_WSTR("base_prefix", config->base_prefix);

+      SET_SYS_FROM_WSTR("exec_prefix", config->exec_prefix);

+      SET_SYS_FROM_WSTR("base_exec_prefix", config->base_exec_prefix);

+ -    {

+ -        PyObject *str = PyUnicode_FromString(PLATLIBDIR);

+ -        if (str == NULL) {

+ -            return -1;

+ -        }

+ -        SET_SYS_FROM_STRING("platlibdir", str);

+ -    }

+ +    SET_SYS_FROM_WSTR("platlibdir", config->platlibdir);

+  

+      if (config->pycache_prefix != NULL) {

+          SET_SYS_FROM_WSTR("pycache_prefix", config->pycache_prefix);

file modified
+11 -1
@@ -17,7 +17,7 @@

  %global prerel b1

  %global upstream_version %{general_version}%{?prerel}

  Version: %{general_version}%{?prerel:~%{prerel}}

- Release: 4%{?dist}

+ Release: 5%{?dist}

  License: Python

  

  
@@ -295,6 +295,12 @@

  # Merged upstream, planned for Python 3.9.0b2

  Patch350: 00350-Fix-sqlite3-deterministic-test.patch

  

+ # 00351 #

+ # Allow overriding sys.platlibdir

+ # See https://bugs.python.org/issue40854

+ # Upstream merge pending, planned for Python 3.10.0-a0

+ Patch351: 00351-Allow-overriding-sys.platlibdir-via-PYTHONPLATLIBDIR.patch

+ 

  # (New patches go here ^^^)

  #

  # When adding new patches to "python" and "python3" in Fedora, EL, etc.,
@@ -651,6 +657,7 @@

  %patch328 -p1

  %patch349 -p1

  %patch350 -p1

+ %patch351 -p1

  

  

  # Remove files that should be generated by the build
@@ -1598,6 +1605,9 @@

  # ======================================================

  

  %changelog

+ * Mon Jun 08 2020 Sandro Mani <manisandro@gmail.com> - 3.9.0~b1-5

+ - Add patch to allow overriding platlibdir

+ 

  * Fri May 29 2020 Petr Viktorin <pviktori@redhat.com> - 3.9.0~b1-4

  - Add cherry-picks for bugs found in 3.9.0b1

  

In mingw-python, I have a mingw{32,64}-python3 wrapper which invokes the native python3 and overrides various variables and paths to point to the ones in the mingw target tree. i.e.:

pylibdynload=$(/usr/bin/python3.9 -c 'import sysconfig; import os; print(os.path.join(sysconfig.get_path("platstdlib"), "lib-dynload"))') \
CC=x86_64-w64-mingw32-gcc \
LD=x86_64-w64-mingw32-ld \
DLLWRAP=x86_64-w64-mingw32-dllwrap \
_PYTHON_HOST_PLATFORM=mingw \
_PYTHON_SYSCONFIGDATA_NAME="_sysconfigdata__win_" \
PYTHONHOME=/usr/x86_64-w64-mingw32/sys-root/mingw \
PYTHONPATH=$PYTHONPATH:/usr/x86_64-w64-mingw32/lib/python3.9:/usr/x86_64-w64-mingw32/lib/python3.9/site-packages:$pylibdynload:/usr/x86_64-w64-mingw32/sys-root/mingw/lib/python3.9:/usr/x86_64-w64-mingw32/sys-root/mingw/lib/python3.9/site-packages \
/usr/bin/python3 "$@"

This is working pretty well, and i.e. a mingw32-python3 setup.py install in most occasions does the right thing.

Now, python-3.9 appears to have introduced sys.platlibdir [1], which breaks the above of the native python is the x86_64 one because i.e. the paths returned by sysconfig.py will include lib64 (so i.e. /usr/x86_64-w64-mingw32/lib64/python3.9:/usr/x86_64-w64-mingw32/lib64/python3.9/site-packages), but mingw always uses lib as libdir.

Following PR adds a method to override the platlibdir via _PYTHON_PLATLIBDIR=lib. I haven't found a less invasive way to override this, since sys is a builtin module.

[1] https://github.com/python/cpython/pull/8068

Build succeeded.

I'd be very much happier if this goes to upstream first. We can backport it.

I agree that this should be put upstream first, otherwise we'll have to keep rebasing this patch forever for each new version of Python.

rebased onto ecf9ebd

3 years ago

https://github.com/python/cpython/pull/20605 was merged upstream. Do you need a backport ASAP, or can his wait for the next update (possibly delayed by the Fedora infrastructure datacenter move)?

Correction, upstream seem to only have this in 3.10, because 3.9 is on feature freeze. Do you need this backported to 3.9 in Fedora?

I'd appreciate if this can be backported downstream for the 3.9 cycle yes - currently mingw-python is pretty broken.

Build failed.

https://bugs.python.org/issue40854 is somehow a Python 3.9 regression and so I consider to include the fix in Python 3.9 upstream: see my https://github.com/python/cpython/pull/20725 backport to Python 3.9.

Closing this since python3.9-3.9.0~b3-1.fc33 is building, which contains this change.

Pull-Request has been closed by smani

3 years ago

Thanks for doing it the hard way!

Closing this since python3.9-3.9.0~b3-1.fc33 is building, which contains this change.

That's great to have it upstream!