Staging
v0.8.1
Revision 85d5c18c9d83a1d54eecc4c2ad4dce63194107c6 authored by R. David Murray on 03 December 2017, 23:51:41 UTC, committed by GitHub on 03 December 2017, 23:51:41 UTC
The original algorithm tried to delegate the folding to the tokens so
that those tokens whose folding rules differed could specify the
differences.  However, this resulted in a lot of duplicated code because
most of the rules were the same.

The new algorithm moves all folding logic into a set of functions
external to the token classes, but puts the information about which
tokens can be folded in which ways on the tokens...with the exception of
mime-parameters, which are a special case (which was not even
implemented in the old folder).

This algorithm can still probably be improved and hopefully simplified
somewhat.

Note that some of the test expectations are changed.  I believe the
changes are toward more desirable and consistent behavior: in general
when (re) folding a line the canonical version of the tokens is
generated, rather than preserving errors or extra whitespace.
1 parent 29ba688
Raw File
pyarena.h
/* An arena-like memory interface for the compiler.
 */

#ifndef Py_LIMITED_API
#ifndef Py_PYARENA_H
#define Py_PYARENA_H

#ifdef __cplusplus
extern "C" {
#endif

  typedef struct _arena PyArena;

  /* PyArena_New() and PyArena_Free() create a new arena and free it,
     respectively.  Once an arena has been created, it can be used
     to allocate memory via PyArena_Malloc().  Pointers to PyObject can
     also be registered with the arena via PyArena_AddPyObject(), and the
     arena will ensure that the PyObjects stay alive at least until
     PyArena_Free() is called.  When an arena is freed, all the memory it
     allocated is freed, the arena releases internal references to registered
     PyObject*, and none of its pointers are valid.
     XXX (tim) What does "none of its pointers are valid" mean?  Does it
     XXX mean that pointers previously obtained via PyArena_Malloc() are
     XXX no longer valid?  (That's clearly true, but not sure that's what
     XXX the text is trying to say.)

     PyArena_New() returns an arena pointer.  On error, it
     returns a negative number and sets an exception.
     XXX (tim):  Not true.  On error, PyArena_New() actually returns NULL,
     XXX and looks like it may or may not set an exception (e.g., if the
     XXX internal PyList_New(0) returns NULL, PyArena_New() passes that on
     XXX and an exception is set; OTOH, if the internal
     XXX block_new(DEFAULT_BLOCK_SIZE) returns NULL, that's passed on but
     XXX an exception is not set in that case).
  */
  PyAPI_FUNC(PyArena *) PyArena_New(void);
  PyAPI_FUNC(void) PyArena_Free(PyArena *);

  /* Mostly like malloc(), return the address of a block of memory spanning
   * `size` bytes, or return NULL (without setting an exception) if enough
   * new memory can't be obtained.  Unlike malloc(0), PyArena_Malloc() with
   * size=0 does not guarantee to return a unique pointer (the pointer
   * returned may equal one or more other pointers obtained from
   * PyArena_Malloc()).
   * Note that pointers obtained via PyArena_Malloc() must never be passed to
   * the system free() or realloc(), or to any of Python's similar memory-
   * management functions.  PyArena_Malloc()-obtained pointers remain valid
   * until PyArena_Free(ar) is called, at which point all pointers obtained
   * from the arena `ar` become invalid simultaneously.
   */
  PyAPI_FUNC(void *) PyArena_Malloc(PyArena *, size_t size);

  /* This routine isn't a proper arena allocation routine.  It takes
   * a PyObject* and records it so that it can be DECREFed when the
   * arena is freed.
   */
  PyAPI_FUNC(int) PyArena_AddPyObject(PyArena *, PyObject *);

#ifdef __cplusplus
}
#endif

#endif /* !Py_PYARENA_H */
#endif /* Py_LIMITED_API */
back to top